Staff Data Scientist, Agent Improvement Loop

Accountabilities:

Own and develop the evaluation strategy for AI agents, including pre-release behavioral testing, production monitoring, and automated evaluation systems.
Conduct continuous analysis of agent traces and conversations at scale to identify recurring failure patterns, determine root causes, and prioritize improvement opportunities based on business impact.
Drive the complete agent improvement cycle by translating analytical findings into actionable fixes, implementing improvements, and measuring their effectiveness through robust before-and-after comparisons and causal analysis.
Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools using evidence gathered from production traces, collaborating with relevant engineering teams and contributing directly to their codebases.
Partner with Delivery and Customer Success teams to improve underperforming customer conversational workflows and establish repeatable processes for maintaining high-quality agent performance.
Design, calibrate, and validate LLM-as-a-judge evaluation systems against human assessments, measuring consistency, reliability, limitations, and overall effectiveness.
Develop and maintain production-quality Python and TypeScript code for evaluation and analytical tools, ensuring appropriate testing, documentation, version control, and CI/CD integration.
Mentor data scientists and engineers in measurement methodologies, experimentation, evaluation practices, and data-driven decision-making.
Contribute to quarterly and annual strategic planning, helping define priorities and long-term initiatives for improving AI agent quality.
Requirements:
Bachelor's or Master's degree in a technical discipline such as Computer Science, Statistics, Applied Mathematics, Operations Research, Economics, or a related field.
At least 8 years of experience delivering data science solutions with measurable business impact in cross-functional environments.
Demonstrated hands-on experience evaluating LLM-based systems or AI agents, including evaluation design, LLM-as-a-judge methodologies, evaluation rubrics, and judge calibration.
Advanced proficiency in Python and SQL, with working knowledge of TypeScript or the willingness to develop expertise in it.
Strong software engineering practices, including version control, code reviews, automated testing, CI/CD, and experience building and deploying tools for shared production environments.
Practical experience working with production GenAI systems, including prompt engineering, tool design, agent frameworks such as LangGraph, and LLM observability platforms such as LangSmith.
Ability to transform complex and unstructured trace and conversational data into actionable insights and communicate findings effectively to both technical and non-technical stakeholders.
Strong product and business acumen, with the ability to connect technical quality improvements to customer experience and measurable business outcomes.
Excellent analytical thinking, problem-solving, communication, and collaboration skills, combined with the ability to work independently and influence cross-functional teams.
Experience with conversational AI, customer-facing agents, or tool-using AI systems is considered a strong advantage.
Knowledge of Natural Language Processing (NLP), clustering techniques, and methodologies for discovering and categorizing recurring system failures is a plus.
Familiarity with causal inference and experimentation methods, including A/B testing, difference-in-differences, and pre/post analysis, as well as the ability to recognize the limitations of available data.
Exposure to MLOps or LLMOps platforms such as Vertex AI, Amazon SageMaker, MLflow, or similar technologies, as well as big data tools, is desirable.
Understanding of AI safety, responsible AI principles, and best practices for developing reliable AI systems is an advantage.
Previous mentoring experience and the ability to support the professional development of other data scientists and engineers are highly valued.
Adaptability and comfort working in an ambiguous, rapidly evolving environment.
Benefits:
Competitive compensation aligned with market standards.
Unlimited paid time off (PTO).
Fully remote working opportunities, with working hours offering approximately three hours of flexibility around Central Time.
Flexible working arrangements focused on results, autonomy, and impact.
Opportunity to work in a dynamic, fast-growing startup environment.
International and collaborative team with opportunities to work alongside professionals from different backgrounds and markets.
Strong opportunities for professional development, career progression, and increased responsibilities.
Inclusive workplace culture that values diversity, initiative, simplicity, excellence, and individual contributions.
An environment that encourages innovation, ownership, and the freedom to bring forward new ideas.