Want to know if this job is worth applying to?
New York, New York, United States
Onsite
At its core, Replay was founded to help founders. We started by supporting startups through shutting down, but we have since expanded into unlocking a new revenue stream for all types of businesses.
In 2025, we had a unique insight: the data every company generates each day through collaboration, communication, and building is some of the most valuable training data in the world. Public and synthetic data can only get frontier models so far, so the next generation of model progress depends on real, proprietary data grounded in how actual businesses operate. We are a primary source of it, partnering directly with the frontier AI labs building what comes next.
We have scaled from $0 to a multi-eight-figure run rate in a matter of months
We have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund
We are small enough that you will carry outsized responsibility and grow as quickly as the company does
You will partner with and build for some of the fastest and most important companies in the world
You will help build a massive, category-defining business from the ground floor
This is a production product-engineering role for someone who has shipped and operated LLM-backed software—not a prompt-engineering, research, or company-wide AI strategy position.
As our Senior AI Product Engineer, you will turn our early dissolution support agent into a trustworthy product that resolves well-bounded customer needs and establishes the right operating model for more complex workflows. You will move between customer experience, application code, retrieval, tool contracts, model behavior, evaluation, permissions, observability, rollout, and production learning. The goal is correct resolution and customer trust—not maximum deflection or autonomy.
Own AI-assisted dissolution support from the customer problem through production behavior, measurement, and iteration
Design retrieval, context, structured outputs, tool contracts, orchestration, and deterministic boundaries for grounded, inspectable behavior
Build clear answer, status, no-action, and human-handoff experiences that remain useful when evidence or authority is incomplete
Create representative, versioned evaluations for routing, grounding, usefulness, safety, stability, and real customer outcomes
Ship with explicit permissions, tenant boundaries, privacy controls, auditability, canaries, rollback, and recovery
Instrument runtime and tool reliability, latency, cost, repeat contact, support effort, and serious failure modes
Turn production failures into durable product, evaluation, and system improvements
Determine whether complex workflows should be automated, AI-assisted, structured for a human, or deliberately remain human-owned, then build the approved product approach
Simplify, replace, or remove agentic components when deterministic software or a clearer product experience would work better
Work closely with Product, Support, Security, domain experts, and full-stack engineers who own the surrounding Dissolution product
Customers get correct, useful resolution for a meaningful set of dissolution needs—not merely fewer human replies
Unsupported claims and unsafe actions remain inside explicit launch guardrails, with sensitive failures treated as stop-ship issues
Human handoffs are timely, accurate, and carry enough context to help the customer rather than restart the conversation
New intents move from evidence design through safe release using repeatable evaluation, tool, observability, and rollout infrastructure
Quality, runtime reliability, latency, cost, privacy, and customer effort remain visible as the product grows
At least one valuable multi-step workflow has an evidence-backed operating model—automated, AI-assisted, or deliberately human-owned—with clear authority, auditability, and recovery
You have personally owned a production software product, including an LLM-backed capability beyond a prototype
You are a strong product engineer who can build across customer experience, application code, backend systems, AI behavior, and production operations
You understand retrieval, context selection, structured outputs, tool use, orchestration, evaluation, and observability—and know when simpler, deterministic software is the better tool
You can turn ambiguous user needs into explicit evidence, state, authority, and failure boundaries
You have designed for unsupported claims, uncertainty, permissions, privacy, human escalation, rollback, and recovery
You use representative evidence to make ship, revise, or stop decisions rather than optimizing demos or one aggregate score
You enjoy learning a consequential domain and working directly with Product, Support, Security, and engineering partners
You use modern AI development tools fluently and verify their output with the same rigor you apply to product behavior
You want to focus primarily on model research, prompt iteration, or AI infrastructure without owning the complete customer and production outcome
You believe more autonomy, more model calls, or a more sophisticated agent framework is inherently better
You prefer to hand off evaluation, security, observability, or production operation after a prototype works
You want a company-wide AI charter rather than focused ownership of the Dissolution product
Experience building customer-support, operations, or multi-step workflow agents
Experience with LangGraph, LangChain, or comparable orchestration approaches
Experience with typed tool protocols, retrieval systems, golden datasets, offline evaluation, shadow deployments, or model-based judges
Experience with privacy-sensitive, multi-tenant, audited, legal, financial, or other high-trust products
Strong Python plus TypeScript, React, Node.js, or comparable full-stack experience
