Technology · Milpitas, CA

Hybrid Full-time

Machine Learning Engineering Technical Leader

Cisco · Milpitas, CA · Posted Oct 23, 2026

This posting comes from Cisco. Confirm the details on their site before applying.

Overview

Cisco’s Canvas AI team consists of AI researchers and software developers who collaborate to build innovative products and platforms for Cisco. We are motivated by tackling unique challenges that arise when connecting people and devices at a world-wide scale. Canvas is an agentic workspace where our customers’ IT teams and our AI agents investigate and resolve issues across every domain.

Responsibilities

  • Own the evaluation roadmap and strategy for agentic AI quality, from architecture to production.
  • Design a layered evaluation architecture: deterministic checks on every PR, model-based evaluation pre-release, and human expert calibration.
  • Build and calibrate multi-model LLM-as-judge panels, tracking true-positive and true-negative rates separately.
  • Run the human-in-the-loop flywheel with SMEs, turning annotations into training signal for automated judges.
  • Score production traffic online, alert on regressions and drift, and convert real traces into offline test cases.
  • Lead simulation, synthetic data, and adversarial testing without consuming lab resources.
  • Close the loop from evaluation failure to clustered issue, candidate fix, and shadow/A/B validation.
  • Ensure trustworthy results through experiment snapshots, dataset versioning, coverage reporting, and cost tracking.
  • Drive common standards, data contracts, and judge criteria across partner teams without direct authority.
  • Mentor engineers, set evaluation engineering standards, and represent the domain to senior leadership.

Requirements

  • BA/BS with 8+ years of relevant experience, MS with 6+ years, or PhD with 3+ years in Computer Science, Machine Learning or a related quantitative field.
  • Experience taking machine learning or LLM-based systems into production and owning them operationally, not only building prototypes.
  • Experience in Python and modern deep learning frameworks such as PyTorch or TensorFlow.
  • Experience designing evaluation, measurement or quality systems for ML or LLM products — datasets, metrics, judges, benchmarks or regression suites.
  • Experience setting technical direction across multiple teams through influence rather than reporting lines.

Preferred qualifications

  • LLM-as-judge design and calibration: practical experience building judge systems and validating them against human experts, including inter-annotator agreement and judge drift.
  • Agentic system evaluation: measuring multi-step reasoning, tool selection, trajectory quality and root-cause correctness rather than single-turn output quality.
  • Production monitoring at scale: asynchronous scoring of live traffic, drift detection, and alerting that engineering teams actually act on.
  • Simulation and synthetic environments: building reproducible test environments or synthetic data generation for domains where ground truth is scarce.
  • Evaluation in CI/CD: integrating automated evaluation into build pipelines and using it to gate releases.
  • Networking, observability or infrastructure domain knowledge: familiarity with the telemetry, topology and failure modes of enterprise network and observability systems.
  • Cost-aware inference: routing, model selection and optimisation techniques that keep evaluation economically viable at production volume.
  • Technical community contribution: publications at top ML venues, open-source contributions, or public technical writing in the evaluation space.
  • 10 paid holidays per full calendar year, plus 1 floating holiday for non-exempt employees
  • 1 paid day off for employee’s birthday, paid year-end holiday shutdown, and 4 paid days off for personal wellness determined by Cisco

Benefits

  • Health insurance, Dental insurance, Vision insurance, 401(k), Paid time off, Paid sick leave, Parental leave, Life insurance, Disability insurance, Bonus, Wellness program

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.