Notion is the collaborative AI workspace where teams and agents think together. We're building one place where your knowledge, projects, meetings, and AI tools live side by side, so work feels faster, clearer, and less fragmented. Millions of individuals, small teams, and large companies run their work on Notion.
Notinos (our employees) are customer zero in bringing this future of work to life. We care about craft, humanity, and building things that last — not just shipping the next feature, but setting a standard for how modern teams (with humans and agents working together) think and execute.
Agent Dev Velocity builds the tooling and evaluation backbone that helps Notion ship high-quality AI faster and more safely. We build the infrastructure that makes AI evaluations easy to create, cheap to run, and hard to ignore, so engineers across the AI org can iterate with confidence.
In this role, you will work at the intersection of developer tooling, distributed systems, and measurement. You will build systems for running and maintaining evals at scale, and you will help create durable benchmarks and datasets that keep us honest about quality over time.
You will help evolve evals into a system, by enabling reusable eval workspaces and data-driven workflows that surface issues through data mining and continuous measurement.
Build and improve scalable eval runners and harnesses that work locally, in CI, and on scheduled runs.
Make it easy for engineers to add high-signal evals: better templates, fixtures, debugging tools, and clear workflows.
Build and maintain benchmark and dataset tooling (curation pipelines, versioning, artifact management, and regression tracking).
Improve reliability and observability for eval execution (retries, idempotency, cost and latency visibility, and failure triage).
Partner closely with AI product, AI platform, and infrastructure teams to integrate evals into day-to-day shipping workflows.
Strong software engineering fundamentals and experience shipping production systems.
Proficiency with TypeScript/Node and/or Python.
Experience building reliable systems in distributed environments (queues, retries, idempotency, and backfills).
Comfort working with data pipelines (batch processing, data quality, versioning, and reproducibility).
Practical experience designing measurement or evaluation systems (LLM eval experience is a plus, but strong testing and benchmarking instincts also apply).
You don’t need to be an AI expert, but you’re curious and willing to adopt AI tools to work smarter and deliver better results.
Experience building developer tooling (CLI tools, CI integrations, or internal platforms).
Familiarity with LLM evaluation techniques (rubrics, human review loops, dataset curation, and regression detection).
Experience collaborating across teams to roll out new workflows and drive adoption.
Notion is committed to providing highly competitive cash compensation, equity, and benefits. The compensation offered for this role will be based on multiple factors such as location, the role’s scope and complexity, and the candidate’s experience and expertise, and may vary from the range provided below. For roles based in San Francisco or New York City, the estimated base salary range for this role is $214,000 - $300,000 per year.
By clicking “Submit Application”, I understand and agree that Notion and its affiliates and subsidiaries will collect and process my information in accordance with Notion’s Global Recruiting Privacy Policy and NYLL 144. #LI-Onsite
You don’t need deep AI expertise for every role, but we do expect every Notino to be intellectually curious, drawn to tinkering and discovery, and excited to use AI as a real collaborator in their work. For some roles, AI fluency is a core requirement — when that’s the case, we’ll make it explicit in the qualifications. People who thrive here don’t treat AI as a novelty. They use it to think better, move faster, and build more creatively.
We hire talented and passionate people from a variety of backgrounds because we want our teams to reflect the wide diversity of our customers. If you’re excited about a role but your experience doesn’t align perfectly with every bullet point listed, we still encourage you to apply.
Notion is proud to be an equal opportunity employer. We do not discriminate in hiring or any employment decision based on race, color, religion, national origin, age, sex (including pregnancy, childbirth, or related medical conditions), marital status, ancestry, physical or mental disability, genetic information, veteran status, gender identity or expression, sexual orientation, or other applicable legally protected characteristic. Notion considers qualified applicants with criminal histories, consistent with applicable federal, state and local law. Notion is also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, please let your recruiter know.