Anthropic’s inference team builds and maintains the critical systems that serve Claude to millions of users worldwide, focusing on maximizing compute efficiency and enabling breakthrough research. The role involves designing intelligent routing algorithms, autoscaling compute fleets, building deployment pipelines, integrating AI accelerator platforms, and supporting new inference features, while contributing to performance tuning and multi-region deployments.
Senior Software Engineer, Inference at Anthropic
Hybrid - London, UK
More jobs at AnthropicSalary
GBP 225,000 - 325,000
Requirements
Skills
- Bachelor’s degree or an equivalent combination of education, training, and/or experience
- A field relevant to the role as demonstrated through coursework, training, or professional experience
- Significant software engineering experience, particularly with distributed systems
- Experience with high-performance, large-scale distributed systems
- Experience implementing and deploying machine learning systems at scale
- Experience with load balancing, request routing, or traffic management systems
- Experience with LLM inference optimization, batching, and caching strategies
- Experience with Kubernetes and cloud infrastructure (AWS, GCP)
- Experience with Python or Rust
- Results-oriented mindset with a bias towards flexibility and impact
- Ability to pick up slack beyond job description
- Interest in machine learning systems and infrastructure
- Thrives in environments where technical excellence drives business results and research breakthroughs
- Care about the societal impacts of work
Responsibilities
- Designing intelligent routing algorithms that optimize request distribution across thousands of accelerators
- Autoscaling compute fleet to dynamically match supply with demand across production, research, and experimental workloads
- Building production-grade deployment pipelines for releasing new models to millions of users
- Integrating new AI accelerator platforms to maintain hardware-agnostic competitive advantage
- Contributing to new inference features (e.g., structured sampling, prompt caching)
- Supporting inference for new model architectures
- Analyzing observability data to tune performance based on real-world production workloads
- Managing multi-region deployments and geographic routing for global customers
Technologies
PythonRustKubernetesAWSGCPLLM inferenceBatchingCachingAI acceleratorsDistributed systemsLoad balancingRequest routingTraffic managementCloud infrastructure
See if your resume is ready for this job
See how our AI can optimize your resume and improve your chances for this role.