Software Development · Menlo Park, CA

Full-time

Staff Software Engineer, Frontier Security Team

Snowflake · Menlo Park, CA · Posted Sep 18, 2026

This posting comes from Snowflake. Confirm the details on their site before applying.

Overview

We are hiring a Staff Software Engineer for our Frontier Security AI team. Snowflake's Frontier Security AI teams develop production-grade LLM applications, intelligent agents, AI infrastructure, and evaluation systems for enterprise customers — products that must meet a high bar for quality, security, reliability, and efficiency while operating over sensitive data at large scale.

Responsibilities

  • Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services.
  • Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review.
  • Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates.
  • Convert ambiguous reports such as "the agent feels worse" into measurable failure modes, reproducible tests, and durable fixes.
  • Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code.
  • Close the loop between production incidents, root-cause analysis, evaluation coverage, and regression prevention.
  • Develop offline and online measurements for task completion, correctness, groundedness, safety, latency, reliability, and cost.
  • Build simulation and replay infrastructure for golden-set tests, adversarial scenarios, model comparisons, and large-scale experiments.
  • Improve agent efficiency through model routing, prompt and semantic caching, context compaction, tool-result management, and token optimization.
  • Productionize new model capabilities as secure, observable, multi-tenant services with clear operational controls.
  • Establish standards for evaluation design, including sampling, ground-truth quality, grader calibration, leakage prevention, and statistical significance.
  • Define technical direction across multiple teams and lead projects whose scope extends beyond a single service.
  • Mentor engineers, raise the quality of architecture reviews, and remain directly involved in implementation and debugging.
  • 9+ years of software engineering experience, including technical leadership of complex production systems.
  • Direct experience shipping and operating LLM applications, AI agents, or model-backed workflows in production.

Requirements

  • Treat evaluation as part of product engineering rather than a final validation step.
  • Can move between agent behavior, distributed infrastructure, data analysis, and production debugging.
  • Question metrics that do not reconcile and design tests that can expose misleading results.
  • Take ownership from early architecture through deployment, operations, and measurable customer outcomes.
  • Prefer evidence from representative tasks and production behavior over isolated benchmark results.
  • Work effectively in fast-moving environments where requirements develop through experimentation.

Preferred qualifications

  • Building evaluation or observability infrastructure for agentic coding, data engineering, or analytics systems.
  • Designing human-evaluation programs, scoring rubrics, annotation workflows, or grader-calibration methods.
  • Working with multi-agent orchestration, long-running agents, asynchronous workflows, or durable execution.
  • Developing synthetic tasks, simulations, adversarial tests, red-team exercises, or safety guardrails.
  • Building retrieval systems that use vector search, hybrid search, semantic indexing, ranking, or caching.
  • Operating multi-tenant systems that process sensitive enterprise data.
  • Working with model training, fine-tuning, reinforcement learning, or feedback-driven optimization.
  • Evaluating and onboarding frontier models based on measured product outcomes.
  • Experience with databases, SQL engines, data platforms, Kubernetes, or cloud-native infrastructure.

Benefits

  • Bonus

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.