Software Development · Sunnyvale, CA

Staff Software Engineer, Inference

CoreWeave · Sunnyvale, CA · Posted May 7, 2026

This posting comes from CoreWeave. Confirm the details on their site before applying.

Overview

As a Staff Software Engineer (IC5) on the Inference team, you will act as a technical leader driving architecture, performance, and reliability across multiple services and teams. Your day-to-day will involve leading cross-team design initiatives, optimizing inference performance (latency, throughput, and GPU utilization), and improving system reliability at scale.

Responsibilities

  • The Inference team builds and operates CoreWeave’s Kubernetes-native inference platform, powering low-latency, high-throughput AI workloads at massive scale.
  • The team is responsible for request routing, scheduling, GPU resource management, and system-wide optimizations that drive performance, efficiency, and reliability across real-time inference systems.

Requirements

  • 8–12+ years of experience building and operating large-scale distributed systems or cloud platforms
  • Proven experience leading cross-team technical initiatives impacting multiple services or organizations
  • Strong programming skills in Go, Python, or C++
  • Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
  • Strong understanding of distributed systems, networking, and performance optimization
  • Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements
  • Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization
  • Experience improving system performance using metrics-driven approaches (e.g., latency, throughput, utilization)
  • Familiarity with mixed precision (BF16, FP8) and streaming inference workloads
  • You love to design and optimize high-performance distributed systems at scale
  • You’re curious about AI inference, GPU systems, and emerging performance techniques
  • You’re an expert in building reliable, low-latency infrastructure and driving system-wide improvements
  • Be Curious at Your Core
  • Act Like an Owner
  • Empower Employees

Preferred qualifications

  • Experience with inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe
  • Experience with GPU systems and performance optimization (CUDA, NCCL, RDMA, NUMA, GPU interconnects)
  • Experience leading multi-team or org-level technical initiatives
  • Exposure to large-scale AI/ML infrastructure or hyperscale cloud environments

Benefits

  • Medical, dental, and vision insurance - 100% paid for by CoreWeave
  • Company-paid Life Insurance
  • Voluntary supplemental life insurance
  • Short and long-term disability insurance
  • Flexible Spending Account
  • Health Savings Account
  • Tuition Reimbursement
  • Ability to Participate in Employee Stock Purchase Program (ESPP)
  • Mental Wellness Benefits through Spring Health
  • Family-Forming support provided by Carrot
  • Paid Parental Leave
  • Flexible, full-service childcare support with Kinside

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.