Software Development · Sunnyvale, CA

Senior Software Engineer, Inference

CoreWeave · Sunnyvale, CA · Posted Sep 26, 2025

This posting comes from CoreWeave. Confirm the details on their site before applying.

Overview

Lead design reviews and drive architecture within the team; decompose multi-service work into clear milestones. Define and own SLIs/SLOs; ensure post-incident actions land and reliability improves release-over-release. Implement advanced optimizations (e.g., micro-batch schedulers, speculative decoding, KV-cache reuse) and quantify impact. Strengthen incident posture: capacity planning, autoscaling policy, graceful degradation, rollback/traffic-shift strategies.

Responsibilities

  • Senior engineers are area owners who lead designs, raise engineering standards, and deliver measurable improvements to latency, throughput, and reliability across multiple services.
  • You’ll partner with product, orchestration, and hardware teams to evolve our Kubernetes-native inference platform and meet strict P99 SLAs at scale.

Requirements

  • 5-8 years industry experience building distributed systems or cloud services.
  • Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems and performance.
  • Optimize end-to-end ML system performance by developing and tuning CUDA kernels, reducing model latency, maximizing compute and memory bandwidth utilization, and leveraging custom accelerators for high-efficiency workloads
  • Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry).
  • Experience with inference internals: batching, caching, mixed precision (BF16/FP8), streaming token delivery.
  • Proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work.

Preferred qualifications

  • Contributions to inference frameworks (vLLM, Triton, TensorRT-LLM, Ray Serve, TorchServe).
  • Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
  • Leading multi-team initiatives or partnering with customers on mission-critical launches.
  • Be Curious at Your Core
  • Act Like an Owner
  • Empower Employees
  • Deliver Best-in-Class Client Experiences
  • Achieve More Together

Benefits

  • Medical, dental, and vision insurance - 100% paid for by CoreWeave
  • Company-paid Life Insurance
  • Voluntary supplemental life insurance
  • Short and long-term disability insurance
  • Flexible Spending Account
  • Health Savings Account
  • Tuition Reimbursement
  • Ability to Participate in Employee Stock Purchase Program (ESPP)
  • Mental Wellness Benefits through Spring Health
  • Family-Forming support provided by Carrot
  • Paid Parental Leave
  • Flexible, full-service childcare support with Kinside

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.