Product Manager Evaluation & Data (EC AI)

As the Product Manager – Evaluation and Data, your mission is to make the quality of Rakuten's AI products measurable, repeatable, and actionable. You will own the evaluation strategy, process, and platform that enable teams to define success, prevent regressions, make confident release decisions, and continuously improve AI experiences through reliable data and insights. 

• High Impact Role: You'll establish how AI product quality is measured, directly influencing release decisions, user experience, and business growth. 

• Evaluation-Driven Culture: You'll turn product requirements and real-world failures into reliable evaluations with talented engineers, data scientists, and domain experts. 

• Ownership & Autonomy: You'll own the evaluation strategy and platform roadmap, making critical decisions on processes, capabilities, quality standards, and priorities across Rakuten's AI products. 

• Collaborative Environment: You'll work closely with product, engineering, data science, analytics, and business teams to define success and achieve shared goals. 

 
Position Details 

• Evaluation Strategy & Criteria: Define success for Rakuten's AI products and translate requirements, user needs, and risks into measurable criteria. 

• Evaluation Quality: Select grading methods, calibrate automated graders with human judgment, and review transcripts and failures. 

• Platform Strategy: Define the platform vision, roadmap, and requirements with engineering and data teams. 

• Platform Capabilities: Prioritize dataset management, model/prompt comparison, automated and human grading, multi-turn evaluation, trace review, dashboards, and release integration. 

• Continuous Improvement: Maintain balanced suites, convert production issues into regression tests, and improve evaluation coverage and accuracy. 

• Adoption & Decisions: Establish governance, enable team and expert contributions, and connect evaluations with production monitoring, A/B testing, feedback, and business KPIs to guide priorities and releases. 

Mandatory Qualifications: 

• Experience: 5+ years of product management experience in AI/ML, evaluation, data products, or platforms. Experience with consumer AI products is preferred. 

• Data Proficiency: Excellent analytical skills and experience working with large datasets.  

• Platform & Technical Acumen: Ability to define platform requirements and communicate model/prompt versioning, tracing, data pipelines, and release integration. 

• Experimentation & Quality: Experience with A/B testing, production monitoring, user feedback, and human evaluation.  

• Communication Skills: Excellent written and verbal skills, with the ability to present complex evaluation results clearly and concisely.  

• Collaboration Skills: Proven ability to lead cross-functional product, engineering, data science, and domain teams.