Machine Learning Engineer, Evaluation
HackerRank
Hybrid in Bangalore, IndiahybridPosted 3 months ago
Skill Required
EngineeringMachine Learning EngineerMachine LearningETLGenerative AIRAG
Key highlights
- Define what rigorous, fair, and meaningful skill evaluation looks like in the AI era
- Build LLM-powered evaluation pipelines at production scale
- Research mindset required to invent new methodologies
- Equal employment opportunity employer
Role overview
HackerRank helps companies like NVIDIA, Amazon, and Microsoft hire and upskill developers based on skills, not pedigree. Our platform is trusted by over 2,500 innovative companies to build strong engineering teams. As software development evolves with AI assistance, the definition of technical talent is changing. This role focuses on solving the challenge of measuring developer skill in an AI-assisted world, where traditional deterministic evaluation (pass/fail test cases) no longer suffices. The opportunity is to define rigorous, fair, and scalable evaluation methodologies for AI-era development, leveraging HackerRank’s unique scale, data, and industry relationships. The role involves building LLM-powered systems to assess AI usage skills at production scale, inventing new evaluation frameworks, and ensuring consistency, fairness, and defensibility across hundreds of thousands of assessments.
Responsibilities
- Build LLM-powered evaluation pipelines that assess AI usage skills consistently, fairly, and at production scale.
- Own the evaluation methodology end to end—what the rubric is, how the model applies it, how you measure whether it is being applied correctly, and how you audit for bias.
- Design and run experiments to determine what good evaluation actually looks like.
- Build RAG pipelines and fine-tuning workflows that make evaluation models adhere reliably to the rules we set for them.
- Define the benchmarking infrastructure: how we know when our evaluation quality has improved, and how we catch regressions before candidates do.
- Translate model behavior into outcomes that product managers, enterprise customers, and candidates can understand and trust.
Requirements
- You have shipped LLM-powered systems in production where consistency and reliability were hard constraints, not nice-to-haves.
- You think as rigorously about how you measure your model as about the model itself. A poorly constructed eval is a worse outcome than a weaker model.
- You have a research mindset. You are comfortable operating in a space where the right methodology does not exist yet and needs to be invented.
- You think in systems. The data pipeline, the model, the serving layer, and the rubric it enforces are one problem to you.
- You can defend ML judgment in plain language to people who are not ML engineers, because the translation layer is part of the job.
Nice to have
- Experience building evaluation frameworks for generative or conversational AI systems.
- Background in educational assessment, psychometrics, or human-in-the-loop evaluation at scale.
- Publications or open-source contributions in LLM evaluation, benchmarking, or alignment.
- Prior work at the interface of research and product, where you had to ship science, not just publish it.
Additional details
- HackerRank is a proud equal employment opportunity and affirmative action employer. We provide equal opportunity to everyone for employment based on individual performance and qualification. We never discriminate based on race, religion, national origin, gender identity or expression, sexual orientation, age, marital, veteran, or disability status. All your information will be kept confidential according to EEO guidelines.
- Our Recruiters use @hackerrank.com email addresses.
- We never ask for payment or credit check information to apply, interview, or work here.
- Want to learn more about HackerRank? Check out HackerRank.com to explore our products, solutions and resources, and dive into our story and mission here.
- Linkedin | X | Blog | Instagram | Life@HackerRank