ML Annotation QA Engineer
Gather AI
IndiaremotePosted 1 month ago
Skill Required
Quality-AssuranceML-Model-QA-AnnotationAnnotation-QualityData-QualityMachine-LearningAI-QA-EngineerData-Annotation-QAAI-QA-AnalystAI-Quality-Assurance-EngineerAI-ML-Testing-EngineerML-Validation-EngineerAI-Quality-Assurance-AnalystAI-Data-Annotation-AnalystData-Annotation-AnalystAI-Data-Annotation-SpecialistMachine Learning EngineerQA EngineerMachine LearningComputer VisionEngineeringStatisticsanalyticalEmbedded CwrittenPythonCloudjQueryJiraSQLandAIFulltime
Key highlights
- 2–5 years of experience in ML QA, annotation quality, data quality, or analytics at an AI/ML company
- BS in Computer Science/Engineering, Electrical Engineering, or equivalent experience
- Experience working with annotated datasets for computer vision/ML and assessing label quality
- Strong root‑cause analysis skills with ability to generate competing hypotheses
- Proficiency with Python, SQL, and Jira for data querying and workflow tracking
- Role centers on ownership of annotation quality, building performance trackers, and driving actionable insights for model accuracy
Role overview
Gather AI is building a vision‑powered platform that uses autonomous drones and existing equipment to digitize warehouse workflows, making facilities smarter, safer, and more efficient. The ML Annotation QA Engineer will own the quality of annotated data for computer‑vision and machine‑learning programs, analyzing data that cannot be reliably outsourced, defining decision rules, and turning annotation output into actionable insight about system performance in the field.
Responsibilities
- Own the judgment‑heavy quality analysis on annotated data that cannot be reliably outsourced — working the daily review queue and producing verdicts and root cause in house.
- Own, version, and refine the verdict taxonomy, decision rules, and quality guidelines for the categories you cover.
- Build and maintain performance trackers over annotated data — error rates by facility, site, equipment, and data format over time, against an agreed baseline.
- Detect anomalies against that baseline and flag them the day they appear rather than weeks later.
- Run root cause analysis on flagged anomalies, distinguishing annotation error from model or system error from genuine degradation in the field.
- Report findings to engineering and ML with reproducible evidence and stated confidence, fast enough that the issue is still observable.
- Identify systematic failure patterns rather than one‑off misses, and maintain a documented pattern library others can use.
- Query and analyse annotation data directly with Python and SQL to test hypotheses, without waiting on extracts from anyone.
- Feed annotation‑quality findings back as concrete SOP and instruction changes when the root cause is labeling rather than system behaviour.
- Specify annotation tool improvements — what the tool should surface so analysis stops requiring manual work — and validate the fixes.
- Stand up quality analysis and reporting for new annotation programs as they come online.
- Track work across Jira and contribute to pre‑release validation for the behaviours you cover.
- Within the first 90 days, take over barcode and location root‑cause analysis from the annotation vendor and own the daily review queue at the agreed threshold.
- Develop working expertise in the annotation pipeline, including data capture granularity, known quality limits, and domain concepts such as racks, locations, LPN, SKU, AWB, OCR vs barcode failure modes.
- Stand up a facility performance tracker with baseline, thresholds, and reporting cadence; establish anomaly detection and earn team trust in its use.
- Investigate flagged anomalies independently, formulate root‑cause hypotheses, and state confidence levels.
- Take ownership of barcode and location verdict taxonomy, decision rules, and the pattern library of known failure modes; version control documentation, close gaps, add edge‑case guidance, and feed changes back to the vendor’s SOPs when labeling is the root cause.
- By six months, establish yourself as the primary owner of annotation quality analysis and root cause, deliver a reliable performance tracker, close the reporting cycle to same‑day, standardize verdict taxonomy and decision rules, build a documented pattern library, improve the annotation tool to reduce manual effort, and create a repeatable method for standing up annotation quality on new programs.
Requirements
- BS in Computer Science/Engineering, Electrical Engineering, or equivalent experience
- 2–5 years of experience in ML QA, annotation quality, data quality, or analytics at an AI/ML company
- Hands‑on experience with annotated ML datasets, including assessing label quality
- Experience working with annotated datasets for computer vision/ML and assessing label quality
- Strong understanding of statistics and ability to work with data
- Root cause analysis – generating competing hypotheses and naming the evidence that separates them
- Familiarity with Python and SQL (or equivalent) for querying and analysing data independently
- Writing quality guidelines, decision rules, and labeling taxonomies
- Excellent documentation, communication, and collaboration skills
- Demonstrated experience finding, diagnosing, and reporting data anomalies to a technical audience
- Ability to reach a defensible answer from unfamiliar data without prior preparation
- Understanding of data privacy and confidentiality requirements when working with customer operational data
- Python
- SQL or an equivalent query language
- Jira
Nice to have
- 2+ years in quality assurance, data quality, or product support
- 1+ years experience with enterprise‑grade ticketing systems (e.g., Jira)
- Computer vision annotation experience as a reviewer or auditor – video event labeling, bounding boxes, polygon segmentation, counting, or classification
- Annotation quality methodology – gold‑set validation, inter‑annotator agreement, sampling design
- Strong spatial and geometric reasoning – relevant wherever labels describe position in physical space
- Experience in warehouse automation, robotics, or computer vision applications
- Dashboarding or BI tooling for recurring reports
- Small team experience
- Experience with annotation platforms and quality tooling
- Experience authoring annotation guidelines or standard operating procedures
Additional details
- About Us: Gather AI is pioneering a new era of warehouse intelligence with a vision‑powered platform that uses autonomous drones and existing equipment to capture real‑time data, digitizing historically manual and error‑prone workflows.
- About the Team: The engineering organization spans autonomy, computer vision, machine learning, embedded and hardware systems, full‑stack, and cloud, working in parallel across multiple active product lines tied to live customer deployments. Ground truth quality sits at the centre of that – the annotated data this role owns is what our models are trained and measured against.
- Success Traits: Analytical, detail‑oriented, comfortable with ambiguity, persistent in chasing causes, resourceful in building processes, strong written communicator, collaborative and respectful cross‑functional partner.
- Originally posted on Himalayas