Staff Software Engineer - Agent Architecture

Responsibilities

We build agentic AI products that our customers interact with across different modalities. These agents sit on top of the same money-movement platform that handles real funds for businesses in regulated industries, so they have to be safe, compliant, and predictable in ways most consumer AI products are not.

We're looking for a Staff Engineer to own the architecture and implementation of these agents end-to-end. This is a builder role for someone who has shipped agents to production at scale, not just used them. You will define how we build agents at PayNearMe: the frameworks, the integration patterns with our existing systems, the guardrails around money and PII, and the testing/eval/observability loop that lets us improve agents safely over time.

Our core platform stack is Ruby on Rails with MySQL (monolith) plus Go microservices on AWS/Kubernetes, with Datadog for observability. Our agents interact with consumers and business partners across a wide variety of use cases.

What you'll do

You'll define and build the agent platform used across applications at PayNearMe—the architectural patterns, the shared infrastructure—and you'll set the bar for how agents are designed, tested, evaluated, and operated in a regulated, money-movement context.

• Own the architectural direction for agentic AI at PayNearMe in partnership with other engineering leaders. We are building an agent platform, not a single agent—our business customers have different rules, brand voices, allowed actions, knowledge bases, and compliance postures, and the architecture has to treat per-tenant configuration, isolation, and evaluation as first-class concerns.
• Produce and maintain architecture documentation (current state, target state, migration plan) and drive alignment across product, engineering, security, and compliance.
• Design, build, and ship production agents—including voice and chat agents for a wide range of payment-related activities—that integrate cleanly with our Ruby on Rails / MySQL platform and partner services (ElevenLabs, Twilio, and others).
• Treat tool design as a first-class discipline: tool schemas, descriptions, idempotency, side-effect semantics, and error surfaces directly determine agent quality, and for money-moving tools they determine whether we can stand behind every action the agent took.
• Make and defend the "what kind of intelligence goes where" decisions: when to lean on a partner's stack vs. orchestrate frontier LLMs directly, when RAG is the right answer vs. tool calls vs. fine-tuning, when a small/fast/cheap model is sufficient vs. when a frontier model is warranted, and where classical ML or deterministic logic is a better fit than an LLM at all.
• Design the seams that let us swap providers, voice vendors, and models as the landscape shifts—without rewriting the agents that sit on top of them.
• Design and operate the agent lifecycle as a closed loop: testing, offline evals, online evals, observability, scoring, and a disciplined feedback path from what we observe in production back into the test suite and eval set.
• Own the rollout discipline for non-deterministic systems: prompt and agent versioning, shadow mode, canary-by-tenant, gradual ramps, and rollback playbooks that account for the fact that the "bad version" may have already taken real payments. The system has to get measurably better over time, not just ship.
• Own the unit economics of agent interactions. Token budgets, prompt and semantic caching, model cascades (cheap model first, escalate on uncertainty), batch APIs, latency-vs-cost tradeoffs, and per-tenant cost attribution should be instrumented and reasoned about explicitly—at scale, the gap between a well-engineered conversation and a naive one is the business.
• Build the guardrails that make agents safe in a payments context: scope enforcement, refusal behaviors, deterministic handoffs for anything money-changing, PCI-compliant handling of card data, PII protection, and clear human-in-the-loop or fallback paths when confidence is low.
• Own the identity and consent model for agent-initiated actions—who the agent is acting as, when step-up authentication is required before a consequential action, and how explicit consent is captured and stored in a form that holds up in a dispute or chargeback.
• Treat prompt injection and social-engineering of the agent as real attack surfaces; stand up a red-team practice that exercises them continuously, especially against money-moving tools.
• Treat voice as its own modality, not a text agent with a microphone—design for latency budgets, barge-in and turn-taking, STT/TTS error modes, DTMF fallback, recording and consent, and the operational realities of telephony.
• Partner with Security, Compliance, and Legal to ensure agent behavior meets PCI-DSS, state-level payments regulations, and our customers' own compliance obligations. Make agent decisions reconstructable: for any consumer interaction we should be able to explain to a regulator, an auditor, or a disputing party exactly why the agent did what it did, on what information, and with what authorization.
• Raise the bar across the org for agent engineering: define shared patterns for prompts, tools, evals, telemetry, and incident response; serve as a reviewer and approver for architecture decision records (ADRs) and major designs in the agent domain.
• Teach the rest of engineering how to build, evaluate, and operate agents—most engineers on the team are picking this discipline up for the first time, and the team's velocity depends on how well that knowledge transfers.
• Partner with the Engineering Managers, Product, and other Staff peers to shape the roadmap—develop deep expertise in both the technical system and the business need (what our customers and their consumers actually want from an agent), and translate that into durable platform capabilities.

Minimum requirements

• 8+ years of software engineering experience, with Staff-level scope (cross-team influence, major initiatives, long-term technical direction).
• Demonstrated experience shipping agentic AI systems to production—not prototypes, not internal copilots, but agents that real users have relied on. You should be able to talk concretely about what broke, what you changed, and how you knew it got better.
• Hands-on experience with at least one modern agent framework (LangGraph, or comparable). You understand the tradeoffs between graph-based orchestration, ReAct-style loops, and more deterministic state machines, and you have opinions about when each is appropriate.
• Deep, lived experience with the full agent lifecycle: prompt and tool design, offline and online evaluation, scoring rubrics, observability and tracing, and the discipline of feeding production signals back into your eval set and test suite.
• Strong system design fundamentals: reliability, consistency, data modeling, and pragmatic API/service boundaries. You can integrate an agent into an existing transactional system without compromising the integrity of that system.
• Comfortable working in a Ruby on Rails / MySQL environment, or confident in your ability to ramp quickly. You don't need to be a Rails expert, but you need to be able to read the code, work with the team that owns it, and design integrations that fit how the platform actually behaves.
• Clear communication and strong judgment in high-stakes, cross-functional environments—especially in conversations with Security, Compliance, and Legal where the right answer is rarely the fastest one.
• Ability to move between high-level architecture and hands-on coding. This role builds.

Preferred qualifications

• Experience shipping voice agents specifically, and a working understanding of how voice differs from text/chat: latency, turn-taking, STT/TTS failure modes, prosody, barge-in, DTMF, telephony quirks, and recording/consent.
• Hands-on experience building and shipping voice-enabled applications — including conversational AI, TTS/STT pipelines, or telephony integrations — using any platform or stack (e.g., ElevenLabs, Twilio, Vonage, Deepgram, Vapi, LiveKit, or similar).
• Experience designing agents that combine multiple styles of intelligence—partner-managed conversational stacks, frontier LLMs, smaller/faster models, RAG, classical ML, and deterministic logic—and choosing among them based on cost, latency, accuracy, and risk.
• Payments/fintech experience, or other regulated/high-integrity domains (healthcare, lending, insurance). Direct experience with PCI-DSS, TCPA, or similar regimes is a strong plus.
• Experience building and operating evaluation infrastructure—LLM-as-judge, rubric-based scoring, regression suites, A/B and shadow testing for agents in production.
• Experience designing agent platforms with multi-tenant configuration—per-customer rules, knowledge, allowed actions, escalation paths, and isolation of evals and guardrails.
• Experience adversarially testing LLM-based systems (prompt injection, jailbreaking, social-engineering of tool-using agents), particularly where the agent can take consequential actions.
• Experience optimizing the unit economics of LLM-based products: caching strategies, model cascades, prompt compression, and per-tenant cost attribution.
• Experience with observability for non-deterministic systems: tracing across LLM calls and tool invocations, capturing the right signals for debugging and eval mining, and turning production traces into test cases.
• Experience integrating AI capabilities into a Ruby on Rails monolith, or evolving a monolith to support a new class of workload without destabilizing it.