Want to know if this job is worth applying to?
Canada
Remote only
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test - Agentic Automation based in Canada.
This is an opportunity for an experienced SDET to help shape how AI agents are tested, evaluated, and trusted in production. You will join a platform engineering team working on a large-scale, cloud-based cybersecurity environment, building the automation that supports reliability, performance, and quality. Alongside traditional functional and integration testing, you will contribute to agentic test generation, AI-powered workflows, and eval-driven development. Your work will help ensure that AI systems behave accurately, safely, and consistently before reaching production. You will collaborate with engineers across the development lifecycle in a remote-first environment that values autonomy, experimentation, and continuous improvement.
Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
Test integrations across platform systems, services, data, applications, and customer-facing features.
Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
Share technical knowledge and contribute to team-wide quality and evaluation standards.
Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
Experience testing web services and APIs, including REST, gRPC, and JSON.
Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.
Nice to have:
End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.
Experience with prompt engineering and designing agent context or tools.
Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.
Annual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.
Market-leading compensation and equity award opportunities.
Comprehensive physical and mental wellness programs.
Competitive vacation allowance and holidays.
Paid parental and adoption leave.
Professional development opportunities available to employees at every level.
Employee networks, local community groups, and volunteer opportunities.
Remote work opportunities across eligible Canadian location
