Software Engineer – AI Evaluation Expert (Contractual)
Patronus AI
IndiaremotePosted 5 days ago
Skill Required
Software-EngineerAI-EngineerMachine-Learning-EngineerEvaluation-EngineerResearch-EngineerAI-Evaluation-EngineerAI-Evaluation-SpecialistAI-Evaluation-ContractorSoftware-Engineering-Evaluation-SpecialistAI-ML-Evaluation-SpecialistML-Evaluation-EngineerAI EngineerSoftware EngineerSoftware DeveloperAIsoftware engineeringSystem Designrelated fieldEngineeringR ProgrammingJavaScriptTypeScriptdevelopingdebugginganalyticsbuildingwrittenPythondesignC++JavaandGenerative AIGoMachine LearningContract
Key highlights
- Salary: $30–$100 per hour (some flat rate tasks)
- Experience: 2+ years professional software engineering, data science or analytics
- Location: Remote (India)
- Type: Hourly, project/task-based contract
- Commitment: Flexible, typically 15–20 hours per week
- Constraint: LLM-generated content is not permitted
Role overview
Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI, founded by former AI researchers and engineers from Meta AI, Amazon AGI, and Google. They are seeking experienced software engineers for task-based contracts to design realistic, challenging tasks for frontier AI models and judge their performance by applying day-to-day engineering judgment to code, reasoning, and technical instructions.
Responsibilities
- Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
- Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
- Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
- Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
- Help improve how AI understands programming, system design, data workflows and engineering best practices.
- Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
- Test and refine content to reflect real-world complexity.
- Review other contributors’ work for accuracy, clarity, and practical relevance.
- Incorporate feedback promptly and flag gaps or unclear guidance.
Requirements
- 2+ years of professional experience in software engineering, data science or analytics.
- Experience with system design, data pipelines, ML or LLM-based applications.
- Prior work in code review, technical writing, AI training or model evaluation.
- Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.
- Experience building, reviewing and debugging production-grade code.
- Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
- Excellent written communication and close attention to detail.
- A bachelor's degree in computer science, engineering or a related field, or equivalent experience.
Additional details
- All tasks and solutions must be human-authored. LLM-generated content is not permitted.
- Tasks must be original and non-searchable.
- Accurate, carefully checked contributions.
- Quality over volume.
- Pay: $30–$100 per hour and some tasks would have a flat rate
- Location: Remote (India)
- Type: Hourly, project/task-based contract (not a full-time role)
- Commitment: Flexible, typically 15–20 hours per week; can be done alongside a full-time job
- Applications are reviewed on a rolling basis. Placement depends on project needs and assessment results.
- Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.
- By clicking ‘Apply’, you agree to Greenhouse's Terms of Service and Privacy Policy.
- By clicking 'Apply', you agree to Patronus AI, Inc. Privacy Policy.
- Originally posted on Himalayas