Posted today · be early
Python Engineer, AI Coding Agent Evaluator
G2i
WorldwideremotePosted 1 day ago
Skill Required
Python-AI-EngineerAI-Python-DeveloperAI-Coding-Agent-EvaluatorPython-EngineerPython-Software-EngineerAI-Evaluation-EngineerCoding-Assessment-EngineerAI-Development-EngineerPython-Automation-EngineerPython-QA-EngineerAI EngineerPython DeveloperPythonAIEngineeringJavaScriptTypeScriptdesignOpenAPIandContract
Key highlights
- $100–$200/hour compensation rate.
- Staff / Principal‑level engineer required (SR+).
- Hands‑on experience with OpenAI Codex, Claude Code, and Cursor.
- Strong background in TypeScript/JavaScript or Python.
- Contract role with 10–20 hrs/week, start ASAP.
- Opportunity to shape evaluation standards for AI‑assisted development tools.
Role overview
We are seeking a senior‑level software engineer (Staff/Principal or equivalent) to evaluate the quality of interactions with modern AI coding agents such as OpenAI Codex and Claude Code. This contract role focuses on assessing whether the model’s responses reflect the judgment and “taste” of a strong engineer, rather than writing production code.
Responsibilities
- Evaluate AI‑generated coding interactions end‑to‑end.
- Judge whether outputs are useful, correct at a high level, and aligned with how a strong engineer would think.
- Assess the quality of explanations and reasoning, not just the code itself.
- Distinguish between different levels of response quality (e.g., what makes something a 2 vs 4).
- Provide clear, opinionated feedback on what worked, what didn’t, and what felt “off” or misleading.
- Help define what great looks like when interacting with tools like Cursor.
- Assess how AI coding agents behave in real‑world scenarios, focusing on sense of response, usefulness of preamble/reasoning, strong engineering judgment, and whether the interaction feels right to an experienced developer.
Requirements
- Staff / Principal‑level engineer (or equivalent experience).
- Strong background in one of the following: TypeScript / JavaScript or Python.
- Hands‑on experience using OpenAI Codex, Claude Code, and Cursor.
- Deep familiarity with modern AI‑assisted development workflows.
- Able to evaluate code without needing to fully execute or deeply review every line.
- Comfortable giving direct, opinionated feedback.
- High bar for what “good engineering” looks like.
Nice to have
- Experience with tools like Cursor or similar AI‑first IDEs.
- Prior exposure to prompt design or evaluation workflows.
- Experience mentoring senior engineers or defining engineering standards.
Additional details
- Contract position paying $100–$200 per hour.
- Estimated commitment of ~10–20 hours per week.
- Engagement runs through early May with possible extension.
- Start date: ASAP.
- Hiring process includes a take‑home evaluation exercise and one behavioral interview.
- Originally posted on Himalayas.
- Check out the Loom video for more details.