Software Development · Santa Clara, CA

Full-time

Software Engineer, Collective Communication and Network Performance

NVIDIA · Santa Clara, CA · Posted Nov 17, 2026

This posting comes from NVIDIA. Confirm the details on their site before applying.

Overview

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It's a unique legacy of innovation that's fueled by great technology—and amazing people. Today, we're tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world.

Responsibilities

  • Develop detection algorithms that identify communication performance problems from always-on telemetry. Distinguish the causes: a late rank, a degraded GPU, fabric congestion, a silent NIC fault, a slower path chosen over NVLink, or a hang.
  • Build the analysis infrastructure these algorithms run on. Ingest per-collective records from many ranks, correlate them with network counters and topology, and produce diagnoses with supporting evidence.
  • Define the data the detectors need from NCCL profiler and tuner plugin interfaces, RDMA transport state, and NIC and switch counters. Collaborate with the owning teams to deliver it at production-acceptable overhead.
  • Reproduce communication problems on internal clusters and verify that automated diagnoses match the known cause. Maintain these cases as regression tests that gate every change.
  • Extend coverage beyond NCCL to point-to-point and other communication paths, guided by evidence from real incidents.
  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
  • 8+ years of relevant software engineering experience, including hands-on work on a collective communication library, an RDMA transport, or network performance for GPU or HPC workloads.
  • Working knowledge of how NCCL or a comparable library operates: algorithms, protocols, channels, transports and topology detection, and how each affects latency and bandwidth.
  • Working knowledge of RDMA networking in practice: InfiniBand or RoCE, NIC and switch counters, congestion control, and the tools used to measure them.
  • Demonstrated experience diagnosing a multi-node communication problem from measurements, proving the cause, and turning that diagnosis into software used by others.
  • Proven history of landing changes in code owned by other teams, from defining requirements to delivery.
  • Strong debugging skills across component boundaries, with changes validated through code review, tests and independent measurements, including when using automated development tools.
  • Proficiency in C++ and Python, and clear communication of evidence, uncertainty and next steps to engineers and users.

Preferred qualifications

  • Contributions to NCCL, UCX, libfabric, NVSHMEM or a comparable communication library, or to their profiling and tuner plugins.
  • Experience with point-to-point and expert-parallel communication paths such as NIXL, DeepEP or NVSHMEM, or with fabrics such as EFA.
  • Experience building detection or root-cause analysis that ran in production, with insight into its precision and recall.
  • Experience operating or debugging training or inference jobs above a thousand GPUs, including hang localization.

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.