ML Annotation QA Engineer

Accountabilities:

• Own the quality analysis of annotated ML and computer vision data that requires in-house judgment, managing daily review queues and producing clear verdicts and root-cause assessments within agreed quality and time thresholds.
• Define, maintain, and continuously refine verdict taxonomies, decision rules, quality guidelines, and annotation standards for the data categories under your ownership.
• Build and maintain performance trackers that measure error rates across facilities, sites, equipment, and data formats over time against agreed baselines and quality thresholds.
• Detect anomalies against established baselines and escalate emerging issues quickly, ensuring problems are investigated while the underlying behavior remains observable.
• Conduct root cause analysis on flagged anomalies, distinguishing between annotation errors, model or system failures, tooling issues, and genuine degradation in field performance.
• Communicate findings to ML, QA, and engineering stakeholders using reproducible evidence, clearly documented reasoning, and stated confidence levels.
• Identify systematic failure patterns rather than isolated errors and maintain a documented pattern library covering known failure modes and recommended responses.
• Use Python and SQL, or equivalent tools, to query and analyze annotation data independently, allowing you to investigate hypotheses without relying on manually prepared data extracts.
• Translate quality findings into concrete improvements to annotation SOPs, instructions, decision rules, and vendor guidance when labeling quality is identified as the root cause.
• Identify opportunities to improve annotation tools and specify functionality that would reduce manual analysis effort, while validating fixes after implementation.
• Establish quality analysis, reporting, and documentation processes for new annotation programs as additional computer vision and machine learning capabilities are introduced.
• Track work through Jira and contribute to pre-release validation for the behaviors, systems, and workflows within your area of responsibility.
• During the first 90 days, take ownership of barcode and location root-cause analysis, become fluent in the annotation pipeline and warehouse data domain, establish a facility performance tracker with agreed baselines and thresholds, and assume primary ownership of the relevant verdict taxonomy and failure-pattern documentation.

Requirements

• Hold a BS degree in Computer Science, Engineering, Electrical Engineering, or a related discipline, or demonstrate equivalent professional experience.
• Bring 2–5 years of experience in ML QA, annotation quality, data quality, analytics, or a closely related function within an AI/ML environment.
• Have hands-on experience working with annotated machine learning or computer vision datasets, including assessing label quality and identifying inconsistencies or anomalies.
• Demonstrate strong statistical and analytical capabilities, with the ability to distinguish meaningful patterns from noise and communicate the limitations of available data.
• Have proven root cause analysis skills, including the ability to develop competing hypotheses and identify the evidence required to determine which explanation is best supported.
• Be comfortable using Python and SQL, or equivalent query languages, to investigate unfamiliar datasets and independently reach defensible conclusions without relying on pre-prepared extracts.
• Have experience creating quality guidelines, decision rules, labeling taxonomies, SOPs, or similar structured documentation used by technical or operational teams.
• Demonstrate excellent written communication skills, as the role requires producing clear reports and evidence that technical stakeholders can act upon.
• Bring strong collaboration and stakeholder management skills, with the ability to work effectively with ML engineers, QA, engineering, and external annotation partners.
• Have a strong understanding of data privacy and confidentiality requirements when handling customer operational data.
• Experience with enterprise ticketing systems such as Jira, computer vision annotation and auditing, gold-set validation, inter-annotator agreement, sampling methodologies, or recurring BI/dashboard reporting is preferred.
• Experience with video event labeling, bounding boxes, polygon segmentation, counting, classification, OCR, or other computer vision annotation workflows is an advantage.
• Experience in warehouse automation, robotics, computer vision applications, or other physical-world AI environments is highly valued.
• Strong spatial and geometric reasoning skills are beneficial, particularly for data describing physical locations, objects, racks, bins, levels, and equipment.
• Be analytical, detail-oriented, persistent, resourceful, comfortable with ambiguity, and willing to present competing hypotheses rather than forcing a conclusion when the data is inconclusive.

Benefits

• Fully remote work from India, offering flexibility to work remotely while collaborating with a technically diverse engineering organization.
• Full-time employment within a multidisciplinary environment spanning machine learning, computer vision, robotics, autonomy, embedded systems, cloud, and software engineering.
• The opportunity to work directly with real-world computer vision and robotics data, with your analysis influencing systems deployed in operational environments.
• Hands-on exposure to ML annotation quality, data analytics, root cause analysis, computer vision, warehouse automation, and AI system performance.
• The opportunity to establish quality frameworks, reporting processes, decision taxonomies, and pattern libraries that can be reused across future AI programs.
• Significant ownership and visibility, with individual contributions directly informing engineering decisions and improvements to production AI systems.
• Close collaboration with ML engineers, QA professionals, software engineers, and other technical stakeholders in a fast-moving environment.
• The opportunity to improve annotation tooling and processes while helping create measurable gains in data quality, operational efficiency, and model performance.