Overview

We are seeking a seasoned techno-functional leader to drive the development and execution of large-scale LLM training programs. This leader would partner with our clients (leading LLM labs) research teams to:

• Identify opportunities for building training datasets to improve model capabilities and performance
• Generate these datasets with high quality and speed
• Build automation tools and processes for scalability
• Deliver the datasets so that they are easily usable by our clients

Key Responsibilities

Operational Leadership & Performance Management

• Lead and scale global delivery teams of 100+, distributed across functions, regions, and levels (ICs, leads, and managers)
• Implement performance management systems that go beyond managerial reporting using data-driven metrics, tools, and products to assess productivity, quality, and output consistency
• Build strong operational structures that allow for transparency, accountability, and early detection of underperformance
• Partner with cross-functional leads to optimize workflows and improve internal tool adoption for delivery efficiency

Data Quality & Scripting-Driven Automation

• Own the quality, accuracy, and scalability of data generated for LLM training
• Move beyond manual QA layers by leveraging Python scripting, APIs, and automation frameworks to measure, validate, and improve dataset integrity
• Design and oversee tools or scripts for data validation, annotation accuracy checks, and pipeline consistency
• Ensure datasets adhere to compliance standards (PII, GDPR, HIPAA) and can be programmatically tested for usability and quality

LLM Training & Evaluation

• Lead generation and delivery of high-quality, scalable datasets focused on SFT, RLHF, reasoning, and agentic workflows
• Oversee the entire data lifecycle from client intake and annotation workflow design to delivery
• Partner with product, research, and engineering teams to implement evaluation metrics (e.g., win rate, inter-annotator agreement, and pairwise preference scoring)

Client Partnership & Communication

• Serve as the primary point of contact for enterprise AI clients; manage expectations, delivery timelines, and escalations
• Build relationships with engineering and research stakeholders by delivering consistently high-quality data
• Communicate effectively across technical and non-technical audiences; provide transparency through structured updates and quality reporting

Team Development & Tooling

• Recruit, mentor, and coach cross-functional leaders (Eng, Data, Ops, and Program Management)
• Drive adoption and improvement of internal tools (e.g., task management systems, quality dashboards)
• Champion continuous improvement across data quality, tools, and delivery processes

Required Qualifications

• 10+ years of experience leading large-scale technical delivery organizations, ideally across AI, ML, or data operations
• Bachelor's degree in Engineering, Computer Science, or equivalent technical discipline
• Demonstrated ability to act as a strategic business partner with our clients, researchers, and engineers at leading LLM labs
• Proven success in building and scaling multi-level high performance teams, with distributed global operations
• Experience managing managers
• Skip-level performance management
• Hands-on technical fluency: ability to write and review data validation scripts
• Demonstrated experience managing dataset generation or annotation for machine learning model evaluation and/or training 
• Familiarity with ML tools and data workflows (e.g., HuggingFace, LangChain, Weights & Biases, Databricks)

Preferred Qualifications

• Experience evaluating large language model performance and/or improving model performance via fine-tuning
• Strong understanding of data quality frameworks, including automation, toolings and manual processes 
• Experience in AI data annotation, model evaluation, and fine-tuning platforms
• Strong communication and storytelling skills with executive stakeholders

Location SF Bay Area (Hybrid)

Compensation: $255,000 to $325,000 OTE + Equity