Applied AI Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Applied AI Engineer based in United States.

We are looking for an Applied AI Engineer to design, build, and test intelligent conversational experiences and AI agents embedded in learner-facing products. You will transform product and instructional requirements into reliable AI behavior across advisory tools, practice simulations, and diagnostic coaching experiences. Working closely with Product, Engineering, Content Studios, and subject-matter experts, you will help ensure AI systems deliver accurate, consistent, and safe interactions. Your work will cover the full development lifecycle, from prompt design and prototyping to evaluation, production deployment, and ongoing monitoring. You will also contribute to the technical safeguards needed to support privacy, human oversight, and accreditation requirements. This is a fully remote, long-term opportunity within an Agile-oriented environment focused on developing innovative learning technologies for regulated professional industries.

Accountabilities:

As an Applied AI Engineer, you will translate complex learning scenarios and product requirements into robust AI workflows. You will take ownership of conversational behavior, integration architecture, testing, and production readiness while collaborating with cross-functional teams to ensure that AI-powered experiences remain reliable and aligned with their intended purpose.

Translate scenario specifications, decision models, and assessment rubrics into prompts, application logic, and testable AI behaviors.

Identify missing requirements, technical constraints, and potential risks, escalating them to Product, Engineering, and subject-matter experts.

Define how AI models interact with application services and approved data sources, including APIs, tools, data structures, validation rules, error handling, and permitted actions.

Design system prompts, structured outputs, and validation mechanisms to detect and prevent invalid responses from reaching learners.

Build conversational runtimes with controlled personas, session state, and clearly defined state transitions.

Separate simulation behavior, assessment logic, and feedback mechanisms to ensure that feedback does not influence assessment outcomes.

Implement retrieval-augmented generation (RAG) and grounding from approved knowledge sources, including source references where required.

Maintain version control and documentation for code, prompts, schemas, and evaluation assets.

Run structured experiments with Product teams, defining clear objectives and measurable success criteria.

Move validated prototypes into production in collaboration with Engineering and verify that deployed behavior remains consistent.

Contribute to technical controls supporting accreditation and compliance requirements, including issue flagging, review workflows, disclosures, and logging.

Monitor and evaluate AI performance, identifying opportunities to improve quality, reliability, security, and efficiency.

Requirements:

The ideal candidate has hands-on experience developing LLM-powered applications and multi-step conversational workflows. You combine strong technical expertise in prompt engineering, agentic systems, retrieval, and evaluation with sound judgment around AI safety, privacy, and human oversight. You are comfortable collaborating with both technical and non-technical stakeholders and can communicate complex trade-offs clearly.

Expert-level prompt and context engineering, including system prompts, few-shot examples, prompt chaining, routing, and breaking complex tasks into manageable steps.

Strong experience with structured outputs, JSON Schema, function calling, validation layers, and retry and fallback mechanisms.

Practical knowledge of RAG architecture, including chunking, metadata filtering, hybrid search, reranking, and citation grounding.

Experience managing token budgets, prompt caching, context compaction, and long-session memory.

Expertise in agentic systems, including tool use, planner/executor patterns, multi-agent orchestration, state machines, and human-in-the-loop checkpoints.

Ability to define tool schemas, permission scopes, and action limits for AI agents.

Experience with AI evaluation methods such as golden datasets, rubric-based scoring, LLM-as-judge calibrated against human reviewers, and regression testing.

Familiarity with LLM observability, tracing, and performance monitoring.

Strong understanding of LLM security risks, including prompt injection, data leakage, jailbreaks, unsafe tool actions, and guardrails.

Ability to compare prompting, retrieval, and model options based on quality, latency, and cost.

Proficiency in Python or TypeScript, including API integrations and automated testing.

Experience with Git-based version control (such as Bitbucket), reproducible tests, and continuous integration.

Comfortable using AI coding agents as part of your standard development workflow.

Strong communication skills, with the ability to explain technical decisions and trade-offs to non-technical colleagues.

Sound judgment regarding privacy, human oversight, and the limitations of AI-generated assessments or advice.

Demonstrated experience building LLM applications with multi-step conversational workflows.

Experience collaborating with subject-matter experts to translate domain knowledge into effective AI behavior.

Preferred Qualifications:

Background in learning technology products, professional assessment, conversational simulations, or regulated industry content.

Benefits:

Fully remote working arrangement.

Long-term professional opportunity.

Opportunity to build and improve AI-powered learning experiences used in regulated professional industries.

Collaborative, Agile-oriented environment with cross-functional teams.

Exposure to advanced LLM applications, conversational agents, and applied AI development challenges.