Python Engineer, AI Coding Agent Evaluator

G2i

WorldwideremotePosted 1 month ago

Skill Required

Python-AI-EngineerAI-Python-DeveloperAI-Coding-Agent-EvaluatorPython-EngineerPython-Software-EngineerAI-Evaluation-EngineerCoding-Assessment-EngineerAI-Development-EngineerPython-Automation-EngineerPython-QA-EngineerAI EngineerPython DeveloperPythonAIEngineeringJavaScriptTypeScriptdesignOpenAPIandContract

Key highlights

$100–$200/hour
Staff / Principal-level engineer
10–20 hrs/week
Experience with OpenAI Codex, Claude Code, and Cursor

Role overview

We’re looking for highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role; you won’t be writing production code, but rather evaluating whether the model thinks like a great engineer, focusing on engineering taste rather than syntax correctness.

Responsibilities

Assess how AI coding agents behave in real-world scenarios
Determine whether the response makes sense
Determine whether the preamble and reasoning are useful
Determine whether the output reflects strong engineering judgment
Determine whether the interaction feels right to an experienced developer
Evaluate AI-generated coding interactions end-to-end
Judge whether outputs are useful, correct (at a high level), and aligned with how a strong engineer would think
Assess the quality of explanations and reasoning, not just code
Distinguish between different levels of response quality (e.g. what makes something a 2 vs 4)
Provide clear, opinionated feedback on what worked, what didn’t, and what felt “off” or misleading
Help define what great looks like when interacting with tools like Cursor
Answer questions such as: Does this feel like something a strong engineer would actually say?, Is this explanation helpful, or just technically correct?, Is the model guiding the user well, or just dumping output?, Would this interaction build or erode trust?
Make subjective but rigorous judgments

Requirements

Staff / Principal-level engineer (or equivalent experience)
Strong background in one of the below: TypeScript / JavaScript, Python
Hands-on experience using: OpenAI Codex, Claude Code, Cursor
Deep familiarity with modern AI-assisted dev workflows
Able to evaluate code without needing to fully execute or deeply review every line
Comfortable giving direct, opinionated feedback
High bar for what “good engineering” looks like