Senior AI Application Engineer

Senior AI Application Engineer

What you will get to do:

RAG Pipeline & Retrieval
• Own the end-to-end RAG pipeline, from data ingestion to response generation.
• Design and implement Azure AI Search indexing strategies, including schema design, scoring profiles, and performance optimization.
• Develop and refine chunking and metadata strategies to maximize retrieval relevance and context quality. 
• Tune retrieval pipelines (hybrid search, vector search, filters, reranking) for accuracy, latency, and cost.
• Build and maintain prompt orchestration frameworks, including dynamic prompt assembly and context injection.
• Ensure strong grounding and citation mechanisms so model outputs are traceable and verifiable. 
Agentic Systems & Integration
• Design and implement AI agents and agentic workflows, including tool use, planning, and multi-step task execution.
• Integrate agents with internal and external systems (APIs, databases, SaaS tools) to enable real-world actions.
• Define and implement evaluation frameworks (offline & online) to measure retrieval quality, agentic performance, and answer correctness.
• Design and enforce guardrails (safety, hallucination reduction, policy compliance) across generation and agentic behavior.
• Collaborate with product, data, and platform teams to integrate AI capabilities into applications.
• Monitor system performance and continuously iterate based on user feedback and metrics.
Internal AI Practice - Use Case Development
• Partner with business and technology leadership to identify, evaluate, and prioritize agentic AI opportunities across Flash's operations.
• Translate business problems into AI system designs - scoping what's feasible, defining the data requirements, and owning delivery. 
• Build repeatable patterns and internal tooling that accelerate future AI development across the organization.
• Contribute to their evolving AI governance and responsible use practices.

What will make you successful:

• Experience building or maintaining RAG systems in production environments.
• Experience with hybrid search (keyword + vector) and reranking models.
• Knowledge of evaluation frameworks (e.g. RAGAS, LLM-as-a-judge approaches)
• Experience integrating agents with enterprise tools (CRM, ERP, ticketing systems, etc.).
• Experience optimizing systems for scale, latency, and cost.
• Strong knowledge of search technologies (Azure AI Search, Elasticsearch, or similar).
• Hands-on experience with LLMs, embeddings, and vector databases.
• Experience designing and deploying AI agents or agentic systems.
• Demonstrated ability to integrate agents with enterprise systems - ERP, CRM, ticketing, or similar operational platforms.
• Understanding of information retrieval concepts (ranking, relevance, recall/precision tradeoffs).
• Experience with prompt engineering and LLM orchestration tools.
• Familiarity with evaluation techniques for generative AI and agent systems.
• Ability to translate ambiguous business problems into concrete AI system designs.
• Ability to balance experimentation with production reliability. 
• Hands-on experience building AI agents and agentic workflows, including tool use, multi-step reasoning, and orchestration across APIs and services.
• Understand how to design systems that can plan, retrieve context, take actions, and operate within defined guardrails in production environments.
• Proven ability to build effective, reliable AI agents that complete real-world tasks safely and accurately.
• Ability to provide measurable improvements in retrieval relevance and system performance.
• Previous experience creating robust guardrails and evaluation systems that catch issues early. 
• Experience building scalable, maintainable architecture integrated into core products. 
• Self-starter that thrives on ownership.
• Must have the current and continuing right to work in the United States of America without restrictions or expirations.