Software Development · San Francisco, CA

Hybrid Full-time

Software Engineer - Inference Platform

Baseten · San Francisco, CA · Posted Oct 5, 2026

This posting comes from Baseten. Confirm the details on their site before applying.

Overview

We're looking for distributed systems engineers and product-minded generalists to build the distributed runtime that powers large-scale LLM inference on Baseten. Our inference platform empowers customers to deploy and operate cutting-edge models with industry-leading performance, scalability, and reliability. It also powers Model APIs, our hosted endpoints for the latest open-source models.

Responsibilities

  • Build the infrastructure and orchestration systems that deploy and run large-scale distributed LLM inference, including routing, autoscaling, scheduling, and runtime management.
  • Design, build, and operate Model APIs, with a focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling, and multimodal serving.
  • Implement platform fundamentals such as API versioning, validation, usage metering, quotas, and authentication.
  • Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks for speed, reliability, and quality. Help set best practices for testing, release automation, and operational excellence.
  • Debug and harden complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads to improve reliability and scalability.
  • Partner with Inference Performance engineers and other teams to make new optimizations broadly available to customers and easy to configure.
  • Own projects end to end, from architecture through deployment, monitoring, and iteration on customer feedback. Along the way, make thoughtful tradeoffs between performance, reliability, operational simplicity, and developer experience.

Requirements

  • Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
  • 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs where reliability, latency, and scale are first-class concerns.
  • A proven track record of owning low-latency, reliable backend services, including rate limiting, auth, quotas, metering, and migrations.
  • Infrastructure instincts with a feel for performance: profiling, tracing, capacity planning, and SLO management.
  • Comfort debugging performance and reliability issues across multiple layers of the stack, from application behavior down to runtime and infrastructure internals.
  • A strong sense of developer experience. You think about how systems are used, not just how they work.
  • Eagerness to learn new languages, frameworks, and systems, and a real interest in inference engineering. Prior ML or LLM experience isn't required, though experience with model serving or inference systems is a plus.
  • Excellent written communication and collaboration skills, including writing clear design docs and working across functions.

Preferred qualifications

  • Experience with or contributions to LLM inference engines and frameworks such as vLLM, SGLang, TensorRT-LLM, TGI, or Dynamo.
  • Deep Kubernetes experience, including operators and custom resources, plus familiarity with service meshes or API gateways.
  • Experience with distributed scheduling, autoscaling, or service orchestration.
  • Experience operating GPU workloads in production.
  • A background in developer-facing infrastructure or APIs, or contributions to open-source infrastructure or ML systems.
  • Familiarity with observability tooling, CI/CD systems, or release automation.

Benefits

  • Competitive compensation, including meaningful equity
  • (U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependents
  • Flexible PTO policy
  • Paid parental leave
  • Fertility and family-building stipend through Carrot
  • (U.S. only) Company-facilitated 401(k)
  • Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.