QA Automation Engineer

Accountabilities:

• Translate product requirements, customer use cases, and complex feature interactions into clear, testable outcomes and acceptance criteria in collaboration with Product and Engineering.
• Design, implement, maintain, and expand automated test coverage using Playwright, including critical user journeys, feature functionality, regressions, API integrations, and end-to-end workflows.
• Develop and maintain a risk-based testing strategy that prioritizes customer impact, feature dependencies, product roadmap priorities, and changing delivery requirements.
• Test conversational AI workflows, including streaming responses, conversation history, file uploads, document retrieval and citations, model selection, tool execution, interruptions, timeouts, and partial failures.
• Evaluate AI response quality by creating representative datasets and scoring criteria covering accuracy, grounding, instruction following, and appropriate responses to unsafe requests while accounting for natural variation in model behavior.
• Integrate automated tests into CI/CD pipelines, investigate flaky tests, maintain isolated test data and fixtures, and provide actionable diagnostics when failures occur.
• Communicate release readiness by documenting defects with reproducible evidence, customer impact, severity, coverage gaps, and residual risks ahead of UAT and production releases.
• Maintain decision history and testing documentation, capturing expected behavior, approved changes, and the reasoning behind testing decisions so intended changes can be distinguished from regressions.
• Extend automation across backend APIs, authentication, billing and token behavior, model routing, AI workflow execution, file parsing, MCP and tool execution, agentic harnesses, passthrough APIs, and provider-facing API compatibility.
• Design AI workflow tests efficiently to minimize unnecessary token consumption, external provider calls, latency, and execution costs without reducing meaningful test coverage.
• Build repeatable security-sensitive validation covering user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, safe error handling, and layered security controls.
• Use AI-assisted testing tools responsibly for test design, generation, triage, and maintenance while critically reviewing generated tests, assertions, and suggested repairs for reliability, reviewability, and deterministic validation.
• Scale and maintain automation infrastructure, fixtures, test data, and supporting tooling as the product surface and engineering organization grow.

Requirements:

• At least 4 years of QA automation engineering experience, with demonstrated ownership of automated testing programs or significant automation initiatives.
• Strong hands-on experience building and maintaining Playwright automation, including fixtures, resilient locators, assertions, network handling, and trace-based debugging.
• Strong programming skills in TypeScript or JavaScript, with experience writing maintainable test code and reviewing changes using Git-based workflows.
• Experience with API testing, CI/CD integration, test isolation, and troubleshooting failures across browser, application, and backend layers.
• Ability to interpret complex requirements, identify edge cases, and balance testing depth, customer risk, and delivery priorities.
• Strong written and verbal communication skills, with the ability to clearly explain defects, uncertainty, tradeoffs, coverage gaps, and release risks to both technical and non-technical stakeholders.
• Experience testing authentication, authorization, permissions, customer-data separation, and other security-sensitive application behavior.
• Familiarity with defense-in-depth concepts and validating that multiple layers of security controls operate together effectively.
• Strong analytical and troubleshooting skills, with the ability to work independently while collaborating effectively within a cross-functional engineering team.
• Ability to obtain a Department of Defense Secret security clearance.
• Experience testing LLM applications, retrieval-augmented generation systems, or AI agents is highly desirable.
• Python experience for API testing, test utilities, test-data generation, or AI evaluation workflows is preferred.
• Experience with enterprise or government platforms and auditable testing evidence is advantageous.
• Familiarity with model gateways, MCP tools, tool-using systems, workflow automation, and no-code/low-code platforms such as Power Automate, Zapier, Make, or n8n is beneficial.
• Experience with major generative AI platforms and APIs, including Google Vertex AI, AWS Bedrock, or Microsoft Azure OpenAI, and models such as OpenAI GPT, Anthropic Claude, or Google Gemini is preferred.
• Experience with CI/CD pipelines such as GitHub Actions, Docker, Kubernetes, and observability or monitoring tools is advantageous.
• Experience with PostgreSQL and SQL for test-data setup, teardown, and validation is desirable.
• Knowledge of government compliance frameworks such as FedRAMP, NIST AI RMF, and CMMC 2.0 is preferred.
• An active DoD security clearance at the Secret level or above is highly desirable.

Benefits:

• Fully remote position for candidates currently residing in the United States.
• Targeted annual compensation range of $95,553–$143,329, with actual compensation determined by factors such as skills, competencies, experience, education, certifications, location, and business needs.
• Opportunity to work at the intersection of QA automation, generative AI, software engineering, cybersecurity, and government technology.
• Direct ownership of automated quality coverage across browser, application, API, backend, and AI infrastructure layers.
• Exposure to advanced AI testing challenges involving LLMs, retrieval-augmented generation, agents, model routing, MCP tools, and AI workflow execution.
• Opportunity to contribute to secure, auditable technology environments with government compliance and security requirements.
• Collaborative work with Product and Engineering teams on complex technical problems and evolving platform capabilities.
• Opportunity to use modern automation, CI/CD, cloud, observability, and AI-assisted testing technologies.
• Remote flexibility combined with opportunities to develop expertise in emerging AI quality engineering practices.