Software Development Engineer in Test - AI

Accountabilities:

AI/LLM Quality Engineering: Design, implement, and optimize testing strategies for AI-powered applications, chatbots, copilots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.
AI Evaluation Frameworks: Develop and maintain AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regressions, output consistency, and safety guardrails.
Automated AI Validation: Build automated validation pipelines for LLM-generated outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics. Create synthetic and benchmark datasets to validate different AI scenarios.
Agentic Systems Testing: Validate multi-agent orchestration workflows, agent planning and execution, memory retention, context management, tool calling, function calling, and structured outputs. Test MCP (Model Context Protocol) integrations and interoperability.
Prompt and Context Engineering: Contribute to prompt engineering, context engineering, and prompt optimization strategies to improve AI response quality, reliability, and consistency.
Performance and Cost Optimization: Evaluate AI application latency, token consumption, throughput, scalability, and model usage costs to ensure efficient system performance.
Automation Engineering: Develop and maintain scalable UI and API automation frameworks using Python, with tools such as Robot Framework, Playwright, Selenium, and Cypress. Build reusable testing libraries, utilities, and AI-assisted testing workflows.
API and Application Testing: Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming services, webhooks, microservices, and micro-frontend architectures.
RAG and AI Platform Validation: Validate RAG architectures, including document chunking strategies, embedding quality, retrieval accuracy, reranking, and vector search relevance. Work with vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.
AI Agent Frameworks: Build, test, and validate AI agents using frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.
DevOps and CI/CD Integration: Integrate automated testing and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI/CD, or Azure DevOps.
Cloud and Infrastructure Testing: Work with Docker, Kubernetes, and cloud environments across AWS, Azure, OCI, and GCP to validate application reliability, scalability, and deployment quality.
Observability and Reliability: Monitor AI application behavior using logging, tracing, and observability platforms. Validate distributed systems reliability, identify performance bottlenecks, and contribute to shift-left quality engineering practices.
Cross-Functional Collaboration: Partner with engineering, AI/ML, product, platform, and DevOps teams to embed quality throughout the development lifecycle and continuously improve testing standards.
Requirements:
Bachelor's or Master's degree in Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, or a related field. Equivalent practical experience and demonstrable technical projects are also highly valued.
Strong programming skills in Python, with the ability to develop automation frameworks, reusable testing utilities, and complex validation workflows.
Solid understanding of software testing principles, quality engineering methodologies, and modern test automation practices.
Hands-on experience with AI/LLM testing, prompt engineering, context engineering, AI evaluation frameworks, RAG architectures, and AI agents.
Practical experience designing and validating AI-powered workflows, including automated output evaluation, hallucination detection, response quality assessment, and safety validation.
Experience with API testing and automation, microservices testing, UI automation, and data validation using SQL.
Familiarity with AI agent frameworks and technologies such as LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP (Model Context Protocol).
Experience integrating automated testing into CI/CD pipelines and working with Docker, Kubernetes, and cloud platforms such as AWS, Azure, OCI, or GCP.
Strong debugging, analytical, troubleshooting, and problem-solving abilities, with the capacity to investigate complex technical issues independently.
Experience working in Agile or Scrum environments and collaborating effectively with multidisciplinary engineering teams.
Demonstrated curiosity about emerging AI technologies, with the ability to rapidly learn and experiment with new frameworks, tools, and architectures.
Strong communication and collaboration skills, combined with a proactive mindset and passion for innovation, automation, and developer productivity.
Good to have:
Experience with Robot Framework and Playwright for scalable test automation.
Hands-on familiarity with AI development tools such as Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs.
Experience working with vector databases, embeddings, and semantic search technologies.
Exposure to MLOps workflows and machine learning lifecycle testing.
Knowledge of performance testing, security testing, and AI governance or compliance requirements.
Experience with contract testing, chaos testing, resilience engineering, and distributed systems validation.
Familiarity with accessibility testing, cross-browser and cross-device testing, synthetic monitoring, and production observability.
Bonus Skills – High Impact:
AI red teaming and LLM security testing, including prompt injection, jailbreaks, and data leakage detection.
Agent reliability, safety, and autonomous workflow testing.
AI model and infrastructure cost optimization.
Distributed systems testing and test data engineering.
Feature flag testing and AI observability or telemetry.
Experience with reinforcement learning feedback workflows and human-in-the-loop evaluation systems.
Knowledge graph integrations and multimodal AI testing involving vision, audio, and text.
Additional Qualities That Stand Out:
Personal or professional AI projects, prototypes, proof-of-concepts, or open-source contributions.
GitHub repositories demonstrating AI automation, intelligent testing frameworks, or agent development.
Experience building AI agents, copilots, or autonomous testing systems.
Technical blogs, presentations, workshops, or other contributions related to AI and software testing.
Experience integrating AI capabilities into software development lifecycle (SDLC) or quality engineering workflows.
Strong interest in combining traditional quality engineering practices with modern AI-native development approaches.
Benefits:
Fully remote working arrangement from India.
Full-time employment with opportunities to collaborate with international teams.
Competitive healthcare coverage, including an HMO benefit of 175K MBL and one free dependent after one year of service.
Generous vacation policy to support work-life balance and personal well-being.
Punctuality bonus and additional employee benefits.
Access to learning and professional development opportunities to support long-term career growth.
Opportunities for mentorship, cross-functional collaboration, and exposure to different areas of the business.
Inclusive and diverse workplace culture that encourages employees to bring their authentic selves to work.
Access to more than 20 employee affinity groups, providing opportunities to connect, network, and engage with colleagues globally.
Opportunity to work with emerging AI technologies, intelligent automation solutions, and next-generation quality engineering practices.