Senior Python Engineer - LLM Code Evaluation
G2i
WorldwideremotePosted 1 month ago
Skill Required
Senior-LLM-EngineerSenior-AI-LLM-EngineerAI-Code-EvaluationLLM-EvaluatorLLM-EvaluationAI-LLM-Software-EngineerAI-LLM-EvaluationSenior-Python-Developer-AI-MLAI-LLM-EngineerSenior-Python-EngineerPython DeveloperPythonGenerative AIEngineeringJavaScriptTypeScriptdesignOpenAPIandAIContract
Key highlights
- $100–$200/hour
- Staff / Principal-level experience
- 10–20 hours/week
- Start ASAP
- Contract through early May
- Tech focus: OpenAI Codex & Claude Code
Role overview
We’re looking for a highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role; you won’t be writing production code, but rather evaluating whether the model thinks like a great engineer, focusing on engineering taste rather than syntax correctness.
Responsibilities
- Evaluate AI-generated coding interactions end-to-end
- Assess how AI coding agents behave in real-world scenarios
- Judge whether outputs are useful, correct (at a high level), and aligned with how a strong engineer would think
- Assess the quality of explanations and reasoning, not just code
- Distinguish between different levels of response quality (e.g. what makes something a 2 vs 4)
- Provide clear, opinionated feedback on what worked, what didn’t, and what felt “off” or misleading
- Help define what great looks like when interacting with tools like Cursor
- Evaluate whether the response makes sense and whether the preamble and reasoning are useful
- Assess whether the output reflects strong engineering judgment and whether the interaction feels right to an experienced developer
- Answer questions such as: Does this feel like something a strong engineer would actually say?
- Determine if an explanation is helpful or just technically correct
- Evaluate if the model is guiding the user well or just dumping output
- Judge if an interaction would build or erode trust
- Make subjective but rigorous judgments
Requirements
- Staff / Principal-level engineer (or equivalent experience)
- Highly experienced software engineer (SR+)
- Strong background in TypeScript / JavaScript or Python
- Hands-on experience using OpenAI Codex, Claude Code, and Cursor
- Deep familiarity with modern AI-assisted dev workflows
- Able to evaluate code without needing to fully execute or deeply review every line
- Comfortable giving direct, opinionated feedback
- High bar for what “good engineering” looks like
Nice to have
- Experience with tools like Cursor or similar AI-first IDEs
- Prior exposure to prompt design or evaluation workflows
- Experience mentoring senior engineers or defining engineering standards
Additional details
- Contract role
- Rate: $100–$200/hour
- 10–20 hrs/week
- Start ASAP (through early May)
- Check out this Loom video for more details!
- Duration: Through early May (with possible extension)
- Process: Take-home evaluation exercise
- Process: One behavioral interview
- Originally posted on Himalayas