Software Engineer, Cybersecurity Benchmarks

Accountabilities:

Design, develop, and maintain cybersecurity benchmarks for evaluating AI capabilities within the organization's evaluation infrastructure, primarily using the Inspect library.
Develop and prototype new benchmark concepts, experiments, and evaluation methodologies to measure emerging AI cybersecurity capabilities.
Maintain and improve existing benchmark suites, ensuring evaluations can be executed reliably and efficiently whenever new AI models are released.
Design, implement, and strengthen secure sandboxing environments to prevent highly capable AI models from escaping their designated evaluation environments.
Monitor emerging developments in AI agents, coding harnesses, and their interactions with software environments to anticipate potential security vulnerabilities.
Apply threat modeling techniques to identify potential risks, guide benchmark design, and inform broader cybersecurity evaluation strategies.
Contribute strategic recommendations on how AI cybersecurity capabilities should be measured, tested, and evaluated.
Integrate with AI providers and configure benchmarks to run effectively within the organization's infrastructure.
Collaborate with researchers, analysts, and fellow engineers to ensure evaluation data and outputs are accurate, insightful, and suitable for integration into research projects and publications.
Help maintain robust evaluation infrastructure and continuously improve the reliability, security, and scalability of benchmarking processes.
Requirements:
Several years of strong software engineering experience, including building, maintaining, and improving complex systems while consistently producing robust, reliable, and maintainable code.
Practical experience designing and implementing software sandboxing solutions capable of defending against highly capable adversaries.
Previous red teaming experience, including ethical hacking and identifying security weaknesses in software systems.
Familiarity with agentic AI systems, AI coding agents, and the coding harnesses used to execute and interact with tools and environments.
Strong problem-solving and creative thinking skills, with the ability to independently develop ideas for new cybersecurity benchmarks, experiments, and evaluation approaches.
Genuine motivation to contribute to independent AI research and the development of rigorous, publicly accessible evaluations of AI capabilities.
Professional-level English proficiency, with the ability to communicate technical concepts clearly and collaborate effectively in an international research environment.
Ability to work independently in a fully remote setting while collaborating with multidisciplinary teams across different time zones.
Nice-to-Have Qualifications:
Previous experience benchmarking AI systems and conducting model evaluations.
Experience designing or managing Capture the Flag (CTF) infrastructure.
Familiarity with using modern AI tools and systems for red teaming and ethical hacking.
Strong understanding of AI coding harnesses, tool calls, and the mechanisms through which AI agents interact with external environments.
Experience using the Inspect AI framework or familiarity with the Inspect library.
Good understanding of current artificial intelligence developments, emerging capabilities, and cybersecurity trends.
Candidates who do not meet every listed qualification but have relevant experience and a strong interest in the role are encouraged to apply.
Benefits:
Annual Salary: Competitive compensation ranging from $150,000 to $350,000, depending on experience and qualifications.
Work Arrangement: Fully remote position, with opportunities to work from many countries.
Flexible Schedule: Full-time employment with flexible working hours, ideally with some overlap with Pacific Time (UTC–8) and Greenwich Mean Time (UTC).
Team Retreats: Opportunity to participate in three team retreats per year. Travel is not mandatory but is preferred.
Research Impact: Opportunity to contribute to influential research and public evaluations shaping the understanding of AI capabilities and cybersecurity risks.
Professional Growth: Work alongside researchers, analysts, and engineers at the forefront of AI benchmarking and cybersecurity research.
Innovation: Freedom to propose and prototype original benchmark ideas and contribute directly to the organization's evaluation strategy.
Meaningful Mission: Help develop independent, trustworthy research that informs policymakers, industry leaders, developers, and society about advances in artificial intelligence.
Application Flexibility: Applications are reviewed on a rolling basis, with the possibility of starting as soon as the selected candidate is available.