Software Development · Sunnyvale, CA

Senior Software Engineer - AI Infrastructure Performance Insights & Observability

CoreWeave · Sunnyvale, CA · Posted Aug 13, 2026

This posting comes from CoreWeave. Confirm the details on their site before applying.

Overview

We're looking for a Senior Engineer to be a driving force on CoreWeave's Benchmarking & Performance team, with a focus on building the performance insights and observability systems that make our AI infrastructure legible at every layer, from individual GPUs and NVLink/InfiniBand fabrics up through distributed training and inference workloads.

Responsibilities

  • Performance Insights & Observability - Design and build the systems that continuously assess AI infrastructure health and performance: GPU utilization and efficiency, interconnect (NVLink, InfiniBand, RoCE) fabric behavior, distributed training and inference throughput, and hardware degradation…
  • Time-Series & Metrics Infrastructure - Own and extend our time-series database (TSDB) layer as the backbone of real-time observability. Write and optimize PromQL/MetricsQL queries that power alerting, anomaly detection, and trend analysis across thousands of GPUs and hundreds of benchmark runs.
  • Fabric & GPU Telemetry - Build and validate the pipelines and metrics that make network fabric and GPU-level behavior observable and comparable across racks, clusters, and hardware generations, including gray failure detection, congestion and error-rate signals, and health scoring that holds up…
  • Data Lake Architecture (in support of insight work) - Design and build the performance data lake that underpins the above: table formats (Apache Iceberg, Parquet, Avro), hot/cold tiering, and schema evolution for latency distributions, throughput metrics, GPU utilization, cost-per-token, and…
  • Query Optimization & Performance - Profile and tune query engines against columnar and time-series stores so that the observability layer meets its own strict P99 latency and freshness SLAs. Benchmark the benchmarking infrastructure itself.
  • BI & Reporting (secondary) - Where needed, build self-service views (Grafana, Looker, or similar) for engineers, product managers, and executives, but as a downstream output of the insight and observability work above, not the primary deliverable.

Requirements

  • 5+ years of experience building distributed systems, observability platforms, or performance engineering tooling, ideally for infrastructure or ML systems rather than general-purpose BI.
  • Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems, GPU infrastructure, and performance analysis.
  • Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry) used to monitor and diagnose infrastructure, not just report on it.
  • Working knowledge of time-series databases and fluency in PromQL or MetricsQL for building real-time alerting and anomaly detection, not only historical dashboards.
  • Familiarity with data lake architectures and modern table formats (Iceberg, Parquet, Avro) sufficient to support an insights platform, though this is not the primary skill this role is hiring for.
  • Comfortable working close to hardware and workload behavior: GPU utilization patterns, interconnect fabric health, distributed training/inference performance characteristics.
  • Strong communicator comfortable collaborating with cross-functional teams and external partners.

Preferred qualifications

  • Experience with time-series databases, LSM-based storage engines, or custom telemetry pipelines.
  • Experience running MLPerf submissions or similar large-scale audited benchmarks.
  • Contributions to OSS projects such as Apache Iceberg, Apache Spark, Trino, llm-d, vLLM, or PyTorch.
  • Direct experience benchmarking or monitoring large GPU fleets or multi-region clusters.
  • Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
  • Familiarity with data cataloging, lineage tools, or data governance frameworks.

Benefits

  • Medical, dental, and vision insurance - 100% paid for by CoreWeave
  • Company-paid Life Insurance
  • Voluntary supplemental life insurance
  • Short and long-term disability insurance
  • Flexible Spending Account
  • Health Savings Account
  • Tuition Reimbursement
  • Ability to Participate in Employee Stock Purchase Program (ESPP)
  • Mental Wellness Benefits through Spring Health
  • Family-Forming support provided by Carrot
  • Paid Parental Leave
  • Flexible, full-service childcare support with Kinside

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.