AI Research Scientist – Scientific Evaluation (ML/AI)

Accountabilities:

As an AI Research Scientist specializing in scientific evaluation, you will critically examine AI-generated research outputs and determine whether they demonstrate the scientific rigor and originality expected at top-tier machine learning and AI conferences. Your responsibilities will include:

• Evaluate AI-generated ML/AI research papers for scientific contributions, technical correctness, methodology, experimental design, and the validity of their conclusions.

• Assess the novelty and significance of proposed approaches by comparing them with existing research and state-of-the-art methods.

• Examine experimental rigor, including baseline comparisons, benchmark selection, evaluation metrics, statistical evidence, and ablation studies.

• Identify methodological flaws, unsupported claims, inconsistencies, misleading results, and other issues that could compromise scientific validity.

• Assess reproducibility and technical soundness by reviewing experimental evidence, implementation details, and accompanying code when available.

• Verify relevant academic literature, identify overlooked prior work, check citation accuracy, and challenge unsupported novelty claims.

• Determine whether research outputs meet the scientific standards associated with leading ML/AI conferences, including NeurIPS, ICML, and ICLR.

• Produce structured, evidence-based evaluations outlining research strengths, weaknesses, critical technical concerns, and overall scientific merit.

• Apply detailed assessment rubrics consistently while providing clear, concise, and well-reasoned justifications for your evaluations.

• Exercise independent expert judgment when determining whether AI-generated research represents a genuine scientific contribution and is supported by sufficient evidence.

Requirements:

The ideal candidate is an accomplished ML/AI researcher with a strong publication record, deep familiarity with current research, and the ability to critically evaluate complex scientific contributions. Candidates must demonstrate the following qualifications:

• A PhD in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical discipline, or equivalent demonstrated research expertise.

• At least three peer-reviewed research publications at NeurIPS, ICML, and/or ICLR. This is a mandatory requirement.

• Ideally, first-author publications or other significant research contributions demonstrating substantial scientific involvement.

• Extensive hands-on ML/AI research experience, including developing novel methods, designing experiments, analyzing results, and publishing original research.

• Strong knowledge of scientific methodology and experimental rigor, including baseline selection, statistical validity, benchmark design, ablation studies, and reproducibility.

• Ability to independently assess the novelty, significance, technical soundness, and scientific validity of advanced ML/AI research.

• Familiarity with state-of-the-art research and the ability to evaluate new approaches against established literature and benchmarks.

• Experience reviewing research papers for leading ML/AI conferences or journals, particularly NeurIPS, ICML, or ICLR; this is strongly preferred.

• Ability to verify technical arguments, experimental evidence, scientific claims, citations, and code implementations where relevant.

• Excellent analytical skills and scientific writing abilities, with the capacity to communicate nuanced technical judgments through clear, concise, evidence-based feedback.

• Ability to apply detailed evaluation guidelines and scoring rubrics consistently while maintaining independent scientific judgment.

• Exceptional attention to detail, particularly when identifying subtle methodological weaknesses, unsupported conclusions, and discrepancies between claims and evidence.

• Willingness to provide a CV, a brief overview of your research background and areas of expertise, and links to at least three qualifying peer-reviewed publications.

• Relevant academic peer-review experience, including service as a reviewer or area chair at major research conferences, should be highlighted where applicable.

Benefits:

• Fully remote opportunity open to candidates worldwide.

• Competitive compensation of $50–$65 per hour.

• Freelance or independent contractor arrangement.

• Part-time, project-based work with flexible hours.

• Opportunity to evaluate cutting-edge autonomous AI systems capable of conducting end-to-end scientific research.

• Chance to apply your research expertise to the evaluation and advancement of frontier AI agents.

• Exposure to emerging developments in AI-driven scientific discovery.

• Opportunity to contribute specialist scientific judgment to research evaluation processes focused on technical validity, originality, and experimental rigor.

• Flexible working arrangements that support independent research professionals.