Senior AI Systems Quality Engineer

Accountabilities:

• Build and deploy production-grade automated validation frameworks, test harnesses, and evaluation pipelines across the full AI development lifecycle.
• Design and evolve an AI testing platform integrated with Databricks and MLflow to enable repeatable testing, traceability, lineage, and auditability.
• Create large-scale, scenario-based test suites covering hundreds or thousands of cases, including edge cases, long-tail scenarios, and system failure modes.
• Validate agentic orchestration behaviors such as tool usage, memory, decision logic, and non-deterministic outputs before production deployment.
• Embed quality-by-design principles by defining system contracts, guardrails, safe-degradation patterns, and validation requirements at key system boundaries.
• Define measurable quality signals for LLM systems, including grounding, hallucination rates, relevance, latency, cost, accuracy, and explainability.
• Integrate automated quality gates into CI/CD pipelines and ensure validation runs continuously following model, prompt, or code changes.
• Build reusable testing libraries, frameworks, and components that enable engineering teams to adopt consistent AI quality practices.
• Establish measurable release-readiness criteria and support go/no-go decisions based on defined quality thresholds.
• Partner with AI, platform, security, and delivery teams to translate business and mission requirements into clear quality criteria, trade-offs, and confidence levels.
• Evaluate system behavior, reliability, security, privacy, and operational risk in regulated and mission-critical environments.

Requirements:

• 7+ years of software engineering experience, primarily focused on backend or platform systems.
• Proven experience designing and implementing automated AI testing and validation solutions in production environments.
• Demonstrated ability to build custom testing, validation, or evaluation frameworks for complex and distributed systems.
• Strong proficiency in Python and/or TypeScript within modern AI engineering environments.
• Hands-on experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behavior.
• Experience designing AI testing at scale, including regression frameworks, long-tail evaluations, and broad test coverage.
• Deep understanding of CI/CD practices and experience embedding automated tests and quality gates into deployment pipelines.
• Solid knowledge of AWS cloud-native architectures.
• Strong track record of engineering for quality, reliability, governance, safety, and operational resilience as core system principles.
• Working knowledge of security, privacy, and operational risk within regulated or mission-critical environments, including failure modes and recovery strategies.
• Experience with AI testing methodologies such as non-deterministic output evaluation, drift detection, bias and fairness testing, and robust regression strategies.
• Ability to establish measurable trust thresholds and operationalize metrics such as query accuracy, hallucination limits, explainability, and PHI-safe behavior as release criteria.
• Experience collaborating with domain experts to define correctness and real-world validation scenarios that reflect genuine production use cases.
• Experience with Databricks and Medallion architecture is preferred but not required.
• Familiarity with MLflow for model evaluation, lineage, and auditability is a plus.
• Exposure to observability tools such as Datadog, Prometheus, or Grafana is desirable.
• Familiarity with LLM evaluation techniques, guardrails, and policy enforcement frameworks is beneficial.
• Experience evaluating AI performance, latency, and cost regressions is a plus.
• Ability to clearly communicate system behavior and quality trade-offs to both technical and business audiences.
• Formal AI/ML training or certifications, such as ISTQB AI Testing, AWS ML Specialty, or Google ML Engineer, are welcome.
• Experience designing prompts, agent behaviors, and orchestration logic as versioned, testable artifacts is advantageous.
• Familiarity with using AI systems to generate and expand diverse, adversarial, and large-scale test scenarios is a plus.

Benefits:

• Compensation based on experience, skills, and location, including base salary, performance bonus eligibility, and equity grants.
• Unlimited paid time off.
• Work-from-anywhere flexibility.
• Comprehensive health coverage with multiple plan options.
• Equity for every employee.
• Growth-focused environment with opportunities for professional development.
• One-time home office setup allowance.
• Monthly cell phone allowance.