Research Scientist – Science of Evaluation

The deadline for applying to this role is February 22 2026, end of day, anywhere on Earth. 

About the Team  

AISI's Science of Evaluation team develops rigorous techniques for measuring and forecasting AI capabilities, ensuring evaluation results are robust, meaningful, and useful for governance. 

Evaluations underpin both scientific understanding and policy decisions about frontier AI. Yet current methodologies are poorly equipped to surface what matters most: underlying capabilities, dangerous failure modes, forecasts of future performance, and robustness across settings. We address this gap by stress-testing the claims and methods in AISI’s testing reports, improving evaluation methods, and building new analytical tools. Our research is problem-driven, methodologically grounded, and focused on impact. We aim to improve epistemic rigour and increase confidence in the claims drawn from evaluation data. 

 Our approach involves:   

(1) Methodological red teaming: Independently auditing evidence and claims in evaluation reports shared with model developers. 

(2) Consulting partnerships: Collaborating with AISI evaluation teams to improve methodologies and practices. 

(3) Targeted research bets: Pursuing foundational work that enables new insights into model capabilities. 

New research agenda focus (in addition to core team responsibilities): 

Frontier agents increasingly use massive inference budgets on complex, long-horizon tasks. This makes measuring model horizons, estimating performance ceilings, and maintaining research velocity harder and more expensive. We're developing evaluation methods that remain informative as task budgets exceed 10M+ tokens per attempt and model horizons surpass the longest available tasks. 

Role Summary 

This research scientist role focuses on evaluation methods for frontier AI, with emphasis on long-horizon agents and inference-compute scaling. 

You'll design and conduct experiments that extracts deeper signal from evaluation data, uncovering underlying capabilities. You'll collaborate with engineers and domain experts across AISI and with external partners. Researchers on this team have substantial autonomy to shape independent agendas, and push the frontier of what evaluations can reveal. 

Example Projects  

• Develop methods to forecast long-horizon performance under increasing inference budgets, including predictive models based on task and model characteristics 

• Design approaches that preserve observability when agents exceed available task lengths (e.g., proxy measurements, task decomposition, data acquisition strategies) 

• Support evaluation suite design for improved coverage, predictive validity, and robustness 

• Engineer tools for quantitative transcript analysis to identify failure modes and capability signals 

Responsibilities  

• Applied research on evaluation methodology, including new techniques and tools 

• Run and analyze evaluation results to stress-test claims, characterize model capabilities, and inform policy-relevant reports 
• Track the state of the art in frontier AI evaluation research across AISI and externally, and contribute to AISI's presence at ML conferences 
• Long-horizon / inference scaling focus:
  • Design and run experiments that are more informative than end-to-end pass/fail metrics
  • Develop and engineer approaches to long-horizon task design, including automation and internal structure (checkpoints, bottlenecks, progress metrics)
  • Estimate capability upper bounds by identifying measurable bottleneck skills relevant to long-horizon performance. 

Person Specification  

We're flexible on exact background and expect successful candidates to meet many (but not necessarily all) criteria below. Depending on experience, we'll consider candidates at Research Scientist or Senior Research Scientist level. We also welcome applications from earlier-career researchers (2–3 years of hands-on LLM experience) who demonstrate creative and rigorous empirical instincts. 

 Essential  

• Strong track record in applied ML, evaluation science, or experimental fields with significant methodological challenges (e.g., PhD in a technical field, publications at top-tier venues (e.g. ICML, NeurIPS), or substantial real-world deployments) 

• Significant hands-on experience with LLMs and agents 

• Strong motivation for impactful work at the intersection of science, safety, and governance 

• Self-directed and adaptable; comfortable with ambiguity in a growing team 

 Nice to Have  

• Task design and validation experience (checkpoints, verifiers, progress metrics) 

• Transcript analysis or behavioural measurement 

• Experimental design or measurement tooling from other disciplines (psychometrics, behavioural economics). 

Core Logistical Requirements 

• You should be able to spend at least 4 days per week on working with us    

• You should be able to join us for at least 18 months    

• You should be able work from our office in London for parts of the week, but we provide flexibility for remote work    

What We Offer 

Impact you couldn't have anywhere else 

• Incredibly talented, mission-driven and supportive colleagues. 

• Direct influence on how frontier AI is governed and deployed globally. 

• Work with the Prime Minister’s AI Advisor and leading AI companies. 

• Opportunity to shape the first & best-resourced public-interest research team focused on AI security. 

Resources & access 

• Pre-release access to multiple frontier models and ample compute. 

• Extensive operational support so you can focus on research and ship quickly. 

• Work with experts across national security, policy, AI research and adjacent sciences. 

Growth & autonomy 

• If you’re talented and driven, you’ll own important problems early. 

• 5 days off learning and development, annual stipends for learning and development and funding for conferences and external collaborations. 

• Freedom to pursue research bets without product pressure. 

• Opportunities to publish and collaborate externally. 

Life & family* 

• Modern central London office (cafes, food court, gym), or where applicable, option to work in similar government offices in Birmingham, Cardiff, Darlington, Edinburgh, Salford or Bristol. 

• Hybrid working, flexibility for occasional remote work abroad and stipends for work-from-home equipment. 

• At least 25 days’ annual leave, 8 public holidays, extra team-wide breaks and 3 days off for volunteering. 

• Generous paid parental leave (36 weeks of UK statutory leave shared between parents + 3 extra paid weeks + option for additional unpaid time). 

• On top of your salary, we contribute 28.97% of your base salary to your pension. 

• Discounts and benefits for cycling to work, donations and retail/gyms. 
   

*These benefits apply to direct employees. Benefits may differ for individuals joining through other employment arrangements such as secondments. 

Salary 

Annual salary is benchmarked to role scope and relevant experience. Most offers land between £65,000 and £145,000 made up of a base salary plus a technical allowance (take-home salary = base + technical allowance). An additional 28.97% employer pension contribution is paid on the base salary. 

This role sits outside of the DDaT pay framework given the scope of this role requires in depth technical expertise in frontier AI safety, robustness and advanced AI architectures. 

The full range of salaries are available below: 

• Level 3: £65,000–£75,000 (Base £35,720 + Technical Allowance £29,280–£39,280) 

• Level 4: £85,000–£95,000 (Base £42,495 + Technical Allowance £42,505–£52,505) 

• Level 5: £105,000–£115,000 (Base £55,805 + Technical Allowance £49,195–£59,195) 

• Level 6: £125,000–£135,000 (Base £68,770 + Technical Allowance £56,230–£66,230) 

• Level 7: £145,000 (Base £68,770 + Technical Allowance £76,230) 

Selection Process   

In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process.   

The interview process may vary candidate to candidate, however, you should expect a typical process to include some technical proficiency tests, discussions with a cross-section of our team at AISI (including non-technical staff), conversations with your workstream lead. The process will culminate in a conversation with members of the senior team here at AISI.   

Candidates should expect to go through some or all of the following stages once an application has been submitted:   

• Initial interview   

• Technical take home test   

• Second interview and review of take home test   

• Final interview with members of the senior leadership team