Posted today · be early
Expert Talent Pool, Evals and Benchmarks (Associate)
Clera
WorldwideremotePosted today
Skill Required
Expert-Evaluations-and-BenchmarkingAI-Benchmark-SpecialistBenchmark-SpecialistAI-EvaluatorEvaluation-EngineerandUser ResearchEngineeringwrittenPythondesignAIMachine LearningFulltime
Key highlights
- Compensation of $30 to $60 per hour ($62.4K - $124.8K range).
- Level Required: Zero to five years of professional experience.
- Key benefit: Fully remote work.
- Notable requirement: Experience writing test cases, exam questions, rubrics, or evaluation tasks.
Role overview
Join a specialist talent pool supporting human-data projects that evaluate AI models. You will help design and run assessments that measure model capabilities and contribute to AI safety and capability research.
Responsibilities
- Write questions and tasks that isolate and test specific model capabilities.
- Develop rubrics that define strong, partial, and poor answers.
- Create reference answers and edge cases to challenge AI models.
- Grade model outputs against rubrics and explain scoring decisions.
- Analyze failed answers, group error patterns, and document findings.
- Compare multiple models using identical prompts.
- Validate benchmarks and flag compromised or ineffective questions.
- Participate in small user studies evaluating ratings or interactions with model outputs.
Requirements
- Zero to five years of professional experience.
- Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing.
- Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
- Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns.
- Clear written communication and the ability to compare AI model outputs or participate in user and rating studies.
Nice to have
- Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo is useful.
Benefits
- Fully remote.
Additional details
- Compensation is $30 to $60 per hour, depending on project, field, and experience.
- Payment is for hours worked.
- Projects typically last from one week to several months.
- Compensation Range: $62.4K - $124.8K.
- Originally posted on Himalayas.