Engineering · Milpitas, CA

Hybrid Full-time

AI Observability Engineering Technical Leader

Cisco · Milpitas, CA · Posted Oct 27, 2026

This posting comes from Cisco. Confirm the details on their site before applying.

Overview

The Platform and Security team within Cisco’s AI Software and Platform Group designs and operates the infrastructure that powers Cisco’s AI powered offerings, including Cisco Cloud Control, AI Defense, and AI Canvas. Within this team, the Observability Engineering group provides the metrics, traces, and logs needed to keep these services observable and ensure the most important workflows are continuously monitored.

Responsibilities

  • You will define the observability platform vision and execute it end-to-end by partnering with software teams to instrument services correctly and consistently.
  • You will deliver 100% coverage of distributed tracing in collaboration with development teams, ensuring tracing is implemented optimally with the right balance rather than tracing “everything.” You will own and evolve OpenTelemetry enablement—including libraries, process documents, and design…
  • You will drive observability best practices across logging standards and cardinality management, then operate the platform components (Splunk Platform/Splunk Observability, OTEL collectors, and telemetry pipelines) to keep them dependable at scale.
  • You will advance an agentic observability outcome so agents can analyze observability health signals and deliver distilled, actionable information more rapidly than conventional monitoring.

Requirements

  • Bachelor’s degree in Computer Science, Engineering (or related field) +12 years working in a platform services or DevOps role, with specific ownership of observability, or Masters + 8 years of related experience, or PHD + 5 years of related experience
  • 2+ years of experience in Splunk Cloud and Splunk Observability (or SignalFx): experience using Splunk Cloud for logging and Splunk Observability for metrics and traces.
  • Experience with distributed tracing concepts and implementation, including instrumentation approaches and strategies for managing scope and overhead.
  • Experience coding in Python and/or Go, including building libraries, automation, or operational tooling that supports observability platforms.
  • Experience with distributed systems infrastructure, including Kubernetes, OpenTelemetry (OTEL) collectors, and infrastructure-as-code (e.g., Terraform), plus familiarity with Helm charts and modern SDLC practices (e.g., GitHub workflows).

Preferred qualifications

  • Experience driving large-scale (100+ microservices) distributed tracing initiatives across an organization.
  • Experience using AI agents to continually refine observability outcomes (e.g., detecting issues, improving routing/alerts, recommending instrumentation changes).
  • Understanding of LLM observability, including familiarity with major providers and patterns for measuring reliability, latency, and quality signals.
  • Deep understanding of AWS services commonly used in telemetry pipelines and distributed architectures.
  • 10 paid holidays per full calendar year, plus 1 floating holiday for non-exempt employees
  • 1 paid day off for employee’s birthday, paid year-end holiday shutdown, and 4 paid days off for personal wellness determined by Cisco
  • Non-exempt employees** receive 16 days of paid vacation time per full calendar year, accrued at rate of 4.92 hours per pay period for full-time employees
  • Exempt employees participate in Cisco’s flexible vacation time off program, which has no defined limit on how much vacation time eligible employees may use (subject to availability and some business limitations)
  • 80 hours of sick time off provided on hire date and each January 1st thereafter, and up to 80 hours of unused sick time carried forward from one calendar year to the next
  • Additional paid time away may be requested to deal with critical or emergency issues for family members

Benefits

  • Health insurance, Dental insurance, Vision insurance, 401(k), Paid time off, Paid sick leave, Parental leave, Life insurance, Disability insurance, Bonus, Wellness program

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.