AI Operations Specialist

POSITION SUMMARY

We are seeking a thoughtful, analytical, and technically curious AI Operations Specialist to help scale and operationalize Agent AI across Customer Care. This is not an engineering role and not a traditional operations role. AI Operations Specialists sit at the intersection of AI systems, frontline support expertise, and operational rigor - ensuring our AI agents deliver accurate, high-quality resolutions at scale.

In this role, you will define how Agent AI behaves in real-world scenarios by creating and maintaining AI Operating Procedures (AOPs/SOPs), evaluating AI performance through structured QA, and driving iteration loops with Product, Engineering, and external vendors. You will translate human support judgment into system-level instructions and validation frameworks, directly influencing containment, accuracy, and customer experience.

You will operate as part of a cross-functional task force with Product and Engineering, with daily standups and tight execution loops. This role reports to Care Innovation leadership, which holds final authority on go/no-go decisions for AI releases.

The US base salary range for this full-time position is $85,000 - $100,000, plus equity and benefits. Our salary ranges are determined by role, level, and location. This is a hybrid position in Mountain View, CA, requiring in-office work 2 days a week.

WHAT YOU'LL DO

AI Model & SOP Development (35%) 

• Design and maintain detailed AI Operating Procedures (AOPs/SOPs) that define how Agent AI should behave across supported use cases.
• Translate human-agent judgment into structured system logic, mapping actions, edge cases, and expected outcomes.
• Inspect and reference APIs, system actions, and integration flows directly within documentation.
• Continuously update SOPs as product functionality and use cases evolve.

AI Quality Assurance and Performance Improvement (35%) 

• Define and execute QA frameworks to evaluate containment, accuracy, and resolution quality across vendor bots, internal bots, and LLM-based systems.
• Identify failure modes, escalation drivers, and judgment gaps.
• Partner with Product, Engineering, and vendors to iterate on performance and improve customer experience.
• Enable and calibrate human QA teams evaluating Agent AI at scale.

AI Analytics and Operational Insights (20%) 

• Build and maintain dashboards tracking key AI metrics (e.g., containment rate, accuracy, escalation rate).
• Analyze trends and surface actionable insights to guide prioritization and iteration.
• Use data to inform readiness decisions and highlight performance risks.

Cross-Functional Execution (10%) 

• Participate in daily standups with Product and Engineering to align on priorities and iteration loops.
• Provide structured operational feedback to vendors and internal stakeholders.
• Support release readiness discussions and post-launch performance reviews.

WHAT WE'RE LOOKING FOR 

• A college degree or equivalent qualification.
• Three to five years of relevant experience in operations, QA, product operations, support tooling, or adjacent roles.
• A curious, proactive, and entrepreneurial mindset with strong problem-solving skills grounded in critical thinking and common sense.
• Demonstrated ability to reason through complex systems and translate human judgment into structured processes.
• Comfort working with APIs, system actions, and technical documentation (without being an engineer).
• Strong analytical skills and experience working with performance metrics and dashboards.
• Willingness to roll up sleeves and actively participate in execution-heavy work.
• Excellent interpersonal skills and empathy for customers.
• Strong organizational skills and comfort leading workstreams involving Product, Engineering, QA, and external partners.
• Willingness to travel internationally: This role requires travel up to once per quarter to BPO sites supporting AI QA operations.

#LI-Hybrid