Software Development · San Francisco, CA

Full-time

Staff Software Engineer, Core Infrastructure

Harvey · San Francisco, CA · Posted Apr 27, 2026

This posting comes from Harvey. Confirm the details on their site before applying.

Overview

As a Staff Software Engineer on the Core Infrastructure team at Harvey, you'll play a critical role in designing and building new infrastructure systems while equally scaling and strengthening our existing infrastructure. Our infrastructure is the foundation that powers every user interaction with Harvey — processing billions of prompt tokens and millions of daily requests across our global legal AI platform.

Responsibilities

  • Design and build scalable, fault-tolerant infrastructure systems that power Harvey's AI platform across multiple cloud regions
  • Own and evolve our multi-cloud infrastructure (Azure, GCP), including Kubernetes orchestration, networking, and container management
  • Lead technical initiatives around observability, incident response, and operational excellence — building systems that enable rapid detection and resolution of issues
  • Architect and optimize our distributed systems for reliability, including load balancing, quota management, and failover mechanisms
  • Partner with Product Engineering and Security teams to ensure our infrastructure is an accelerant, not a constraint
  • Drive infrastructure-as-code practices using tools like Terraform and Pulumi to enable reproducible, auditable deployments
  • Mentor engineers and raise the technical bar across the organization through code reviews, design reviews, and technical leadership
  • Design and implement a next-generation model proxy architecture that routes millions of daily inference requests while maintaining model API compatibility and enabling seamless model integration
  • Build distributed rate limiting and quota management systems using Redis-backed algorithms to handle bursty traffic patterns without degrading user experience
  • Architect multi-region deployment strategies that meet strict data residency requirements for global enterprise customers
  • Develop comprehensive observability infrastructure with granular SLA monitoring, burn rate alerts, and detailed token attribution for cost tracking
  • Lead the evolution of our CI/CD pipelines to improve developer velocity while maintaining production stability

Requirements

  • 10+ years of experience in Infrastructure Engineering or Platform Engineering in a production environment
  • Long track record building and scaling complex, large-scale distributed systems
  • Deep proficiency with cloud infrastructure platforms (Azure preferred; GCP or AWS experience transfers well)
  • Strong fluency in Infrastructure as Code (IaC) tools — Terraform, Pulumi, or CloudFormation
  • Solid understanding of Kubernetes, container orchestration, networking, and cloud security at scale
  • Experience with observability tools (Datadog, Sentry) and incident response practices (PagerDuty, Incident.io)
  • Strong programming skills in Python, Go, or similar languages
  • Excellent problem-solving skills, a "spidey sense" of where things could go wrong, and a commitment to operational excellence

Preferred qualifications

  • Experience building infrastructure for AI/ML workloads or high-throughput inference systems
  • Background with distributed rate limiting, load balancing, or quota management systems
  • Experience operating multi-tenant platforms with strict security and compliance requirements
  • Track record of leading complex cross-functional projects and delivering measurable impact

Benefits

  • $236,000 - $290,000 USD

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.