AI - Software Development Engineer in Test (AI - SDET)

Accountabilities:

AI and LLM Quality Engineering
Design, implement, and optimise testing strategies for AI-powered applications, copilots, chatbots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.
Develop AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regression, consistency, and safety.
Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
Create synthetic and benchmark datasets to support comprehensive AI testing scenarios.
Test multi-agent orchestration workflows, including agent memory, context retention, planning, and execution.
Validate Model Context Protocol (MCP) integrations and interoperability between AI models and external tools.
Apply Context Engineering, Prompt Engineering, and prompt optimisation techniques to improve AI reliability and output quality.
Evaluate AI system performance, including latency, token consumption, cost efficiency, throughput, and scalability.
Automation Engineering
Develop, maintain, and scale UI and API automation frameworks using Python, preferably with Robot Framework and tools such as Playwright, Selenium, or Cypress.
Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming applications, webhooks, microservices, and Micro Frontend (MFE) architectures.
Build reusable automation libraries, testing utilities, and AI-assisted quality engineering workflows.
Integrate automated testing into CI/CD pipelines and implement quality gates to improve software delivery reliability.
AI Engineering and Platform Validation
Validate RAG architectures by assessing document chunking strategies, embedding quality, retrieval accuracy, re-ranking, and vector search relevance.
Work with vector databases and search technologies, including Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.
Build, test, and validate AI agents using frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.
Test advanced AI capabilities, including tool calling, function calling, agent memory, planning and execution, multimodal workflows, and structured outputs.
DevOps, Observability, and Reliability
Build and maintain automated quality workflows integrated with CI/CD platforms such as GitHub Actions, Jenkins, GitLab CI/CD, and Azure DevOps.
Work with Docker and Kubernetes environments to validate application deployment, scalability, and reliability.
Perform cloud-based testing and deployments across AWS, Azure, Oracle Cloud Infrastructure (OCI), and Google Cloud Platform (GCP).
Validate distributed systems, focusing on reliability, resilience, and performance under different operating conditions.
Monitor AI system behaviour using observability, logging, and distributed tracing tools.
Promote shift-left testing practices and contribute to continuous improvement in quality engineering standards.
Requirements:
Technical Experience and Core Skills
Strong programming skills in Python, with the ability to develop and maintain robust automation frameworks and testing solutions.
Solid understanding of software testing principles, quality engineering methodologies, and modern software development practices.
Hands-on experience with AI and LLM testing, Prompt Engineering, Context Engineering, AI evaluation frameworks, RAG systems, and AI agents.
Experience building and maintaining CI/CD pipelines and working with Docker, Kubernetes, and cloud platforms such as AWS, Azure, OCI, or GCP.
Strong expertise in API testing and automation, microservices testing, and UI automation.
Experience with SQL, database testing, and data validation.
Strong debugging, analytical, and troubleshooting abilities, with a structured approach to solving complex technical challenges.
Familiarity with Agile and Scrum methodologies and experience collaborating within cross-functional engineering teams.
Preferred Qualifications and Additional Skills
Experience with Robot Framework and Playwright for automated testing.
Knowledge of AI development frameworks and ecosystems, including LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and Model Context Protocol (MCP).
Practical experience using AI development assistants and platforms such as Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs.
Familiarity with vector databases, embeddings, and semantic search technologies.
Exposure to MLOps workflows and machine learning lifecycle testing.
Knowledge of performance testing, security testing, and AI governance and compliance requirements.
Experience with contract testing, chaos engineering, or resilience testing.
Familiarity with accessibility testing, cross-browser and cross-device testing, synthetic monitoring, and production observability.
Innovation and Professional Mindset
Demonstrated interest in experimenting with AI tools and developing personal or professional projects, prototypes, proofs of concept, or open-source contributions.
Strong problem-solving skills and the ability to combine traditional quality engineering principles with modern AI-native development approaches.
Passion for innovation, developer productivity, automation acceleration, and intelligent testing solutions.
Ability to quickly learn and adapt to emerging AI frameworks, technologies, and architectures.
A proactive, curious, and collaborative mindset, with a willingness to explore new approaches and continuously improve engineering practices.
Portfolio and Additional Advantages
GitHub repositories showcasing AI, software automation, or quality engineering projects.
Experience developing AI agents, copilots, or intelligent automation solutions.
Contributions to open-source AI or automation ecosystems.
Technical blogs, presentations, workshops, or other content related to AI and software testing.
Experience integrating AI capabilities into Software Development Life Cycle (SDLC) or Quality Engineering (QE) workflows.
Exposure to autonomous testing systems and self-healing automation frameworks.
Education
Bachelor's or Master's degree in Computer Science, Engineering, Artificial Intelligence, Machine Learning, or a related discipline is preferred.
Equivalent practical experience, technical expertise, and demonstrable projects will also be highly valued.
Bonus Skills
AI Red Teaming and LLM security testing, including prompt injection, jailbreaks, and data leakage detection.
AI agent reliability, safety, and behavioural testing.
AI infrastructure and model cost optimisation.
Distributed systems testing and test data engineering.
Feature flag testing and AI observability and telemetry.
Reinforcement Learning from Human Feedback (RLHF) and human-in-the-loop evaluation workflows.
Knowledge graph integrations and multimodal AI testing involving vision, audio, and text.
Benefits:
Flexible working arrangements: Choose between remote, hybrid, or office-based working environments, depending on the applicable arrangements.
Work-life balance: A culture that recognises the importance of personal wellbeing and encourages a healthy balance between professional and personal commitments.
Healthcare coverage: Access to healthcare benefits, including OSDE 310 coverage, subject to local eligibility and applicable arrangements.
Financial benefits: Monthly salary reviews and an annual bonus programme.
Additional paid time off: Extra F5 days off to support personal time and flexibility.
Learning and development: Tuition reimbursement, LinkedIn Learning access, and in-company English lessons to support continuous professional growth.
Internet reimbursement: Financial support towards internet expenses.
Employee referral programme: Bonus opportunities for successful employee referrals.
Global exposure: Opportunities to contribute to international projects and collaborate with multicultural teams.
Career development: Access to mentoring, new challenges, cross-functional learning opportunities, and opportunities for career progression.
Inclusive workplace culture: A diverse and welcoming environment that celebrates individual perspectives and encourages employees to bring their authentic selves to work.
Employee communities: Access to more than 20 affinity groups, providing opportunities to connect, network, and engage with colleagues worldwide.
Innovation-driven environment: Opportunities to work with cutting-edge AI technologies, autonomous systems, and next-generation engineering tools while helping shape the future of AI Quality Engineering.