Software Development · Santa Clara, CA

Full-time

Senior System Software Engineer, DGX Cloud AI Infrastructure

NVIDIA · Santa Clara, CA · Posted Nov 19, 2026

This posting comes from NVIDIA. Confirm the details on their site before applying.

Overview

NVIDIA’s DGX Cloud team builds the infrastructure behind large-scale AI training for NVIDIA’s own training workloads and for our customers. We build services, tools and open-source software that make ML workloads more efficient and resilient. We’re looking for a Senior Software Engineer for the NVIDIA Recovery and Resilience team. You’ll own our AI infrastructure services, tools and open-source projects to support resilient and efficient training from design through production.

Responsibilities

  • Drive the design, build, launch and operation of our services, tools and open-source projects.
  • Plan and assign the team’s work, mentor junior engineers and review designs.
  • Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA’s own training workloads and customer workloads. This includes supercomputers at major cloud providers and on-premises clusters.
  • Build APIs that connect to NVIDIA’s resiliency systems: telemetry, break/fix automation, and checkpoint and execution systems.
  • Build a job submission layer that hides infrastructure failures from users and restarts long-running training jobs automatically.
  • Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures and recommending fixes.
  • Drive our open-source work. Release components so customers and the community can use them, and work with them to improve those components.
  • Set up how the team runs its services: support, monitoring, alerts, CI/CD, integration test gates and on-call plans.
  • BS, MS or PhD in Computer Science or a related field (or equivalent experience), plus 12+ years of experience.
  • Proven experience who has taken large systems from design through production.
  • Strong experience with Kubernetes and/or Slurm at scale.
  • Hands-on experience with PyTorch, TensorFlow, JAX or Ray.
  • Strong understanding of NVIDIA GPUs and networking, including how they fail.
  • Experience running production services, including CI/CD, monitoring, alerting and on-call.

Preferred qualifications

  • Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
  • Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
  • Background in building external services or maintaining open-source projects in ML infrastructure.

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.