Lead Product Manager, Agent Runtime & Evals

Accountabilities:

• Own the vision, product strategy, and roadmap for the agent runtime platform, enabling non-technical users to create, test, publish, deploy, monitor, improve, and retire AI agents and automations without requiring traditional engineering support.

• Lead the development of agent lifecycle management capabilities, including version control, deployment processes, runtime management, observability, and continuous improvement.

• Define and evolve evaluation and model governance frameworks, including golden datasets, data labeling, automated evaluation, regression testing, and quality gates to ensure automations meet required standards before production.

• Establish evaluation mechanisms for AI agents that assess other agents, ensuring quality, reliability, and measurable performance across large-scale automation environments.

• Shape agent memory, learning, and self-improvement capabilities, ensuring that agents can retain relevant context and improve through experience while delivering measurable user value.

• Lead the development of a model-agnostic platform, defining model routing, cost attribution, and pricing strategies that allow teams to select appropriate models for different use cases.

• Define integration strategies that enable agents to securely access internal systems, business data, and third-party applications through reusable connectors, credentials, permissions, and access controls.

• Establish agent identity and execution models, including user-based permissions and independent service identities, while ensuring appropriate governance and auditability.

• Develop oversight and monitoring experiences that allow business teams to manage large fleets of agents through dashboards, execution histories, traces, metrics, logs, audit trails, and emergency controls.

• Partner with workflow orchestration product leadership to define agent registries, structured contracts, and integration standards that allow agents to operate reliably within governed workflows.

• Collaborate with engineering and design teams to create intuitive configuration tools that enable business specialists to build, test, and optimize automations independently.

• Monitor emerging AI technologies, managed agent platforms, orchestration frameworks, and automation solutions to inform strategic build-versus-buy decisions and partnership opportunities.

• Provide product leadership across adjacent teams, guide complex technical and strategic decisions, mentor product colleagues, and establish high standards for product execution and customer experience.

Requirements:

• 7+ years of product management experience in B2B or enterprise SaaS environments, including at least 3 years working in AI-focused product roles.

• Proven experience owning complex products end to end, from product vision and strategy through roadmap development, execution, launch, and continuous improvement.

• Experience building and scaling AI agents or automation solutions that replace manual business processes, ideally in a zero-to-one product environment.

• Strong practical understanding of AI evaluation methodologies, including golden datasets, data labeling, automated judges, regression testing, performance monitoring, and human-in-the-loop evaluation.

• Ability to work closely with engineering teams on technical concepts such as agent-to-workflow contracts, structured outputs, deterministic execution, agent memory, and model orchestration.

• Strong understanding of how non-technical users build, configure, deploy, and supervise automated systems, with a focus on usability and operational reliability.

• Hands-on fluency with modern AI tools and large language models, including platforms such as Claude and Cursor, with the ability to independently prototype solutions and accelerate product development.

• Ability to make informed decisions about AI models, evaluation frameworks, integrations, access controls, and deployment strategies.

• Experience with managed agent platforms such as Claude Managed Agents or Workato Genies is a strong advantage, particularly regarding evaluations, connectors, permissions, versioning, and deployments.

• Excellent written and verbal communication skills, with the ability to explain complex product decisions to engineers, designers, executives, and customers.

• Demonstrated leadership and ability to influence multiple teams, establish shared product direction, and mentor or manage other product managers.

• Strong product judgment, analytical thinking, customer empathy, and comfort navigating ambiguity in a rapidly evolving technology environment.

• Ability to balance technical complexity with practical business outcomes, ensuring that platform capabilities translate into meaningful improvements for end users.

Benefits:

• Competitive compensation: US salary range of USD $200,000–$230,000+, with final compensation depending on experience, skills, and location. Canadian compensation details can be confirmed with the recruiter.

• Equity: Opportunity to participate in the company's long-term growth through equity compensation.

• Comprehensive healthcare: Medical, dental, and vision coverage for employees and their families.

• Life insurance: Financial protection and additional security.

• Mental wellness: Coverage and resources to support mental health and overall well-being.

• Family support: Fertility benefits and support for growing families.

• Flexible time off: Flex Time Off in addition to company-paid holidays.

• Paid leave: Family leave, medical leave, and bereavement leave policies.

• Retirement planning: Access to retirement savings plans.

• Remote work support: Allowance to personalize your home office and technology setup.

• Professional development: Annual stipend to support continued learning, training, and career growth.

• Flexible work environment: Remote-first collaboration with opportunities to work alongside talented colleagues across multiple locations.