Software Development · San Francisco, CA

Full-time

Software Engineer, Spark Platform

DoorDash USA · San Francisco, CA · Posted Oct 13, 2026

This posting comes from DoorDash USA. Confirm the details on their site before applying.

Overview

The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve.

Responsibilities

  • Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling.
  • Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement that let a small team serve dozens of consumer teams.
  • Own pieces of cluster lifecycle automation — provisioning, upgrades, capacity changes, and node-failure handling — at a scale where these stop being manual events.
  • Build the observability and incident automation that make the platform debuggable end-to-end and keep on-call sustainable as the team and the workload grow.
  • Partner with senior engineers on shuffle, runtime, and architecture work, and grow into deeper ownership of those areas over time.

Requirements

  • B.S., M.S., or PhD in Computer Science or equivalent.
  • 2+ years of industry experience operating production distributed systems.
  • Experience operating Apache Spark at scale on Amazon EMR, Databricks, or an in-house deployment — with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi-tenant scheduling) rather than authoring individual Spark jobs.
  • Hands-on experience operating production systems on Kubernetes — controllers, operators, custom resources, and the failure modes that show up in multi-tenant clusters.
  • Familiarity with batch or big-data schedulers (YuniKorn, Volcano, Kueue, or equivalent) and/or with the Spark-on-Kubernetes operator.
  • Familiarity with observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging) and with defining SLOs and SLIs that change team behavior.
  • Comfort working in a cloud environment (AWS preferred) — VPC networking, instance lifecycle, spot/preemptible markets, and autoscaling primitives.
  • Professional experience with Python, Go, Scala, or Java; SQL fluency.
  • A bias toward incremental rollout, measurement, and reducing toil.
  • You are located or willing to relocate to the Bay Area, Seattle, or NYC.

Benefits

  • For salaried roles: flexible paid time off/vacation, plus 80 hours of paid sick time per year.
  • For hourly roles: vacation accrued at about 1 hour for every 25.97 hours worked (e.g. about 6.7 hours/month if working 40 hours/week; about 3.4 hours/month if working 20 hours/week), and paid sick time accrued at 1 hour for every 30 hours worked (e.g.

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.