Posted today · be early
Senior Python Engineer - LLM Code Evaluation
G2i
WorldwideremotePosted 1 day ago
Skill Required
Senior-LLM-EngineerSenior-AI-LLM-EngineerAI-Code-EvaluationLLM-EvaluatorLLM-EvaluationAI-LLM-Software-EngineerAI-LLM-EvaluationSenior-Python-Developer-AI-MLAI-LLM-EngineerSenior-Python-EngineerPython DeveloperPythonGenerative AIEngineeringJavaScriptTypeScriptdesignOpenAPIandAIContract
Key highlights
- $100–$200/hour
- Staff / Principal-level engineer (or equivalent experience)
- 10–20 hrs/week
Role overview
We are looking for a highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role where you will be writing production code; instead, you will be evaluating whether the model thinks like a great engineer, focusing on engineering taste rather than just syntax correctness.
Responsibilities
- Evaluate AI-generated coding interactions end-to-end
- Judge whether outputs are useful
- Judge whether outputs are correct (at a high level)
- Judge whether outputs are aligned with how a strong engineer would think
- Assess the quality of explanations and reasoning, not just code
- Distinguish between different levels of response quality (e.g. what makes something a 2 vs 4)
- Provide clear, opinionated feedback on what worked
- Provide clear, opinionated feedback on what didn’t
- Provide clear, opinionated feedback on what felt “off” or misleading
- Help define what great looks like when interacting with tools like Cursor
Requirements
- Staff / Principal-level engineer (or equivalent experience)
- Strong background in TypeScript / JavaScript or Python
- Hands-on experience using OpenAI Codex
- Hands-on experience using Claude Code
- Hands-on experience using Cursor
- Deep familiarity with modern AI-assisted dev workflows
- Able to evaluate code without needing to fully execute or deeply review every line
- Comfortable giving direct, opinionated feedback
- High bar for what “good engineering” looks like
Nice to have
- Experience with tools like Cursor or similar AI-first IDEs
- Prior exposure to prompt design or evaluation workflows
- Experience mentoring senior engineers or defining engineering standards
Additional details
- Contract role
- Rate: $100–$200/hour
- Hours: 10–20 hrs/week
- Start ASAP (through early May)
- Check out this Loom video for more details!
- Duration: Through early May (with possible extension)
- Process: Take-home evaluation exercise
- Process: One behavioral interview
- Originally posted on Himalayas