Engineering · San Francisco, CA

Hybrid Full-time

Training Performance Engineer

OpenAI · San Francisco, CA · Posted Oct 16, 2025

This posting comes from OpenAI. Confirm the details on their site before applying.

Overview

Training Runtime designs the core distributed machine-learning training runtime that powers everything from early research experiments to frontier-scale model runs. With a dual mandate to accelerate researchers and enable frontier scale, we’re building a unified, modular runtime that meets researchers where they are and moves with them up the scaling curve.

Responsibilities

  • Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
  • Optimize GPU utilization and throughput for large-scale distributed model training.
  • Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
  • Implement model graph transforms to improve end to end throughput.
  • Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
  • Partner with researchers to ensure new model architectures scale efficiently during pre-training.
  • Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.

Requirements

  • Love optimizing performance and digging into systems to understand how every layer interacts.
  • Have strong programming skills in Python and C++ (Rust or CUDA a plus).
  • Have experience running distributed training jobs on multi-GPU systems or HPC clusters.
  • Enjoy debugging complex distributed systems and measuring efficiency rigorously.
  • Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
  • Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.

Preferred qualifications

  • Familiarity with NCCL, MPI, or UCX communication libraries.
  • Experience with large-scale data loading and checkpointing systems.
  • Prior work on training runtime, distributed scheduling, or ML compiler optimization.

Benefits

  • Relocation assistance

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.