As a Research Lead on the Training Insights team at Anthropic, you will develop the strategy for measuring and characterizing model capabilities across training and deployment. The role involves driving original research into new evaluation methodologies, leading a small team of researchers and research engineers, and shaping the evaluation narrative for model releases. You will work across Reinforcement Learning, Pretraining, Inference, Product, Alignment, and Safeguards to map evaluation coverage, identify gaps, and inform training and deployment decisions. Your work will span the full lifecycle of model development, from long‑horizon evaluations to cross‑organizational audits, and you will contribute to the broader research community through publications and open‑source efforts.
Research Lead, Training Insights at Anthropic
Hybrid - Remote-Friendly US (Travel Required) – San Francisco, CA; New York City, NY
More jobs at AnthropicSalary
USD 850,000 - 850,000
Requirements
Skills
- Significant experience designing and running evaluations for large language models or similar complex ML systems
- Led technical projects or teams, either formally or through sustained ownership of critical research directions
- Comfortable designing experiments and writing code, moving between research and implementation fluidly
- Strategically think about what to measure and why, not just how to measure it
- Synthesize information across multiple teams and workstreams to form a coherent picture of model capabilities
- Communicate complex technical findings clearly to both technical and non-technical audiences
- Results-oriented and thrive in fast‑paced environments where priorities shift based on research findings
- Care deeply about AI safety and want your work to directly influence how capable AI systems are developed and deployed
- Experience building evaluations for long‑horizon or agentic tasks
- Deep familiarity with Reinforcement Learning training dynamics and how model behavior changes during training
- Published research in machine learning evaluation, benchmarking, or related areas
- Experience with safety evaluation frameworks and red‑teaming methodologies
- Background in psychometrics, experimental psychology, or other measurement‑focused disciplines
- Track record of communicating evaluation results to inform high‑stakes decisions about model development or deployment
- Experience managing or mentoring researchers and engineers
Responsibilities
- Build new novel and long‑horizon evaluations
- Develop novel measurement approaches for understanding how model capabilities emerge and evolve during RL training
- Lead strategic evaluation coverage across the company
- Shape the evaluation narrative for model releases
- Lead and mentor a small team of researchers and research engineers, setting research direction and fostering a culture of rigorous, creative research
- Design evaluation frameworks that balance scientific rigor with the practical demands of production training schedules
- Build and maintain relationships across Anthropic's research organization to ensure evaluation insights inform training and deployment decisions
- Contribute to the broader research community through publications, open‑source contributions, or external engagement on evaluation best practices
Technologies
Large Language ModelsReinforcement LearningMachine Learning
See if your resume is ready for this job
See how our AI can optimize your resume and improve your chances for this role.