Software Development · Pleasanton, CA

Hybrid Full-time

Software Development Engineer(Distributed Systems)

Workday · Pleasanton, CA · Posted Oct 24, 2026

This posting comes from Workday. Confirm the details on their site before applying.

Overview

The Data Platform and Observability Engineering (DPOE) team is building Workday’s next-generation, multi-petabyte-scale Observability Platform. We are a distributed team responsible for the libraries, distributed services, and infrastructure that power observability across Workday. Our stack includes Iceberg, ClickHouse, Tempo, Grafana, S3, Kafka, Elasticsearch, and LangSmith for LLM/agentic tracing and evaluation.

Responsibilities

  • Develop the Platform: Write high-quality, well-tested code to build and scale features for the distributed tracing platform on ClickHouse/Tempo, focusing on robust ingestion and fast query execution.
  • Maintain Big-Data Pipelines: Build and support high-throughput data pipelines using Kafka, Spark/Flink, and Iceberg-on-S3.
  • Tune and Optimize: Write optimized code and participate in profiling and resolving latency or throughput issues in production environments.
  • Ensure Reliability: Implement robust error handling, retries, and failover mechanisms to ensure high availability for tracing services.
  • Secure the Data: Apply necessary security controls to ensure multi-tenant data access is properly handled according to platform architecture.
  • Operational Support: Maintain the health of the platform through comprehensive monitoring, logging, and alerting, and participate in the team’s on-call rotation.
  • Support Observability AI: Build reliable data ingestion paths that will serve as the foundation for AI-driven anomaly detection and root-cause analysis.
  • Collaborate and Learn: Write clear technical documentation, participate actively in system design reviews, and work closely with Senior and Principal engineers to level up your distributed systems knowledge.

Requirements

  • Strong ability in Algorithmic to build highly efficient and scalable solutions for complex high-throughput data ingestion and sub-second query performance challenges.
  • Solid experience in API Development, including an understanding of gRPC, REST, and OpenTelemetry (OTLP), with practical experience designing and building scalable distributed APIs for observability data.
  • Strong understanding of Code Testing methodologies, such as distributed load testing and integration testing, and experience contributing to end-to-end telemetry pipeline testing and CI/CD automation.
  • Solid understanding of Distributed Systems Software principles, including data partitioning, eventual consistency, and fault tolerance mechanisms, with hands-on experience in Kafka, Spark, Flink, or ClickHouse.
  • Experience implementing and maintaining High Availability strategies for critical distributed systems, including multi-AZ deployments, robust retry mechanisms, and automated failover.
  • Practical experience with Large Scale Data Processing technologies and frameworks such as Apache Kafka, Spark, Flink, and Apache Iceberg within complex distributed architectures.
  • Good understanding of Large Scale Systems design principles, including distributed data sharding, replication, and query optimization, and experience working on observability pipelines or data lake platforms.
  • Strong understanding of Object-Oriented Design (OOD) principles and architectural patterns for building highly scalable and maintainable distributed systems.
  • Experience with Source Control Management (SCM) tools such as Git and GitHub/Bitbucket, and following best practices for collaborative distributed development workflows.
  • Strong understanding of System Security principles and best practices relevant to securing distributed environments, including mutual TLS (mTLS), multi-tenant authorization (authz), and data encryption.
  • Proven ability to actively collaborate within and across distributed software development teams and contribute constructively to architectural discussions and system designs.
  • Strong skills in creating Technical Writing Documentation for runbooks, system design specs, and API documentation related to distributed systems architecture and design.

Benefits

  • Stock / equity, Bonus, Commission, Flexible schedule

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.