Manager, AI Evaluation

• Full-time continuing opportunity
• drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub
• Base Salary $152,045+ 17% superannuation 

About the opportunity

As the Manager, AI Evaluation, you will lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment and ongoing use. You will establish and oversee a consistent evaluation framework across the AI Hub, driving pre-release testing, governance approvals, performance measurement and continuous monitoring of AI services in production. Working closely with product, delivery and technical teams, as well as key stakeholders across ICT, Legal, Privacy and data stewardship functions, you will play a critical role in embedding best-practice evaluation processes, identifying risks, and ensuring AI solutions continue to deliver reliable and high-quality outcomes across the University.

Your key responsibilities will be to:

• lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio
• design, curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions
• oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions
• manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement
• partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met
• provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community

About you

• postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
• demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
• strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
• highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making
• experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
• excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations

Sponsorship / work rights for Australia

You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment.

Pre-employment checks and declarations 

  Your employment is conditional upon the successful completion of all pre-employment or background checks required for the role in terms satisfactory to the University.  Also, to meet the University’s obligations under the National Higher Education Code to Prevent and Eliminate Gender-Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender-based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment.

EEO statement 

At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+. We welcome applications from candidates from all backgrounds. 

We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion. 

How to apply

Applications (including a cover letter, CV, and any additional supporting documentation) can be submitted via the Apply button at the top of the page.  

For employees of the University or contingent workers, please login into your Workday account and navigate to the Career icon on your Dashboard.  Click on USYD Find Jobs and apply.

For a confidential discussion about the role, or if you require reasonable adjustment or any documents in alternate formats, please contact Victoria Ford-Rogers Recruitment Operations by email to victoria.ford-rogers@sydney.edu.au

© The University of Sydney

The University reserves the right not to proceed with any appointment.

Click to view the Position Description for this role.

Applications Close

Sunday 30 August 2026 11:59 PM