E

Etched

Designs chips and servers for running AI models

Site Reliability Engineer - Fleet Command & Control

Site Reliability EngineeringConfirmed live in the last 24 hours
Full-timeSan Jose, CA, USA

About the Job

Requirements

  • At least 2 years of experience in site reliability, production engineering, or infrastructure for large-scale production systems; the San Jose lead seat requires 8 or more years.
  • Experience defining alerting standards adopted by multiple teams, using Prometheus, Grafana, or similar tools, and knowledge of service-level objectives, error budgets, and alert design.
  • Real on-call experience for services with external users, including leading incidents and writing blameless postmortems.
  • A track record of debugging production incidents in systems the candidate did not write, under time pressure, using logs, metrics, traces, and core dumps.
  • Experience with application and scripting languages such as Python, Go, or Rust.
  • Hands-on experience with Linux, containers, Kubernetes, and infrastructure as code such as Terraform or Ansible.
  • Fluency with Linux debugging tools including perf, strace, gdb, dmesg, journalctl, lspci, and tcpdump.
  • Comfort working with hardware in the loop, including PCIe device failures, ECC errors, power and thermal events, and failures across software and physical layers.
  • Sound judgment about when automation should act autonomously and when it should page a person.
  • Clear writing and the ability to get five or more teams to agree on shared standards.

Responsibilities

  • Define healthy inference-system signals across sites, racks, nodes, and chips; refine them as the fleet changes; and determine which recovery actions run automatically and which trigger a human page.
  • Define service-level objectives with each team for latency, error rate, and availability; alert on user-facing symptoms; and verify that alert metrics measure what they are intended to measure.
  • Reduce alert noise by suppressing expected alerts during deploys and maintenance, correlating related alerts into incidents, and retiring pages that receive no action.
  • Turn runbooks into agentic hardware-health recovery workflows, including service restarts, node drains, traffic shifts, and spare swaps, with scoped permissions, rate limits, blast-radius checks, and audit trails.
  • Classify events by domain, page the appropriate domain SRE with context, and dispatch on-site technicians for physical work such as tray swaps, reseats, and leak response.
  • Coordinate with Supercomputing and Infrastructure on the telemetry pipeline and key fleet signals; set the OpenTelemetry tracing standard; and report uptime and reliability trends to inform engineering and capacity decisions.
  • Own severity levels, incident channels, communications, postmortems, and corrective-action tracking; design sustainable on-call rotations, handoffs, and onboarding.
  • Review service designs with service owners for failure modes and graceful degradation before release; partner with Supercomputing on canaries, rolling node-by-node deploys, and graceful drains to prevent downtime.
  • Share on-call in a follow-the-sun rotation with SREs in other time zones.
  • Acknowledge incidents, assess user impact, open incidents, and decide whether to restart, drain, roll back, or escalate.
  • Trace latency or throughput regressions through serving-stack stages including queueing, prefill, decode, handoffs, and cache to identify bottlenecks.
  • Triage hardware errors on live racks, including PCIe device failures, firmware panic resets, chip-to-chip link errors, HBM ECC errors, and power faults; run recovery tools and decide when to remove a node from service.
  • Use OpenTelemetry distributed traces and correlate deploys, firmware events, host and BMC logs, network data, and facilities telemetry to distinguish software bugs, hardware faults, and coolant or power events.
  • Investigate discrepancies between alerts and dashboards, including capped metrics, stale exporters, and failed scrapes.
  • Direct on-site technicians through reseats, PSU and tray swaps, and leak response with exact steps.
  • Provide a timeline, evidence, and current hypothesis when handing incidents to a domain SRE or the next shift.
  • Turn manual fixes into runbooks and then into tasks executable by agents so recurring incidents do not repeatedly page staff.

Desired Qualifications

  • Experience operating GPU, accelerator, or high-performance computing fleets.
  • Experience with large language model inference serving, including prefill and decode, KV cache, batching, and latency metrics such as time to first token.
  • Network debugging from switch to edge, including tracing issues across switches, load balancers, and Cloudflare, and using edge controls such as rate limits to shed or shift traffic during incidents.
  • Experience with out-of-band management using BMC, Redfish, or IPMI, and fleet-wide firmware rollouts.
  • Experience with data-center telemetry, including power distribution, liquid cooling, CDUs, and DCIM.
  • Experience with PagerDuty or similar incident tooling, including event correlation and automated remediation.
  • Experience working with colocation providers and remote-hands technicians.
  • Experience founding or scaling an SRE function at a startup.

About the Company

Etched

Designs chips and servers for running AI models

Etched designs AI chips and the server racks built around them for AI companies and cloud providers, focusing only on inference, the work of running trained models to answer requests. In 2024, it unveiled a chip hard-wired to run only transformers, the type of AI model behind ChatGPT, after betting at its founding that they would dominate AI. Etched develops its chips, racks, and software together, and each rack links many chips through a shared pool of fast memory so very large models with long inputs can answer quickly. It earns money by selling these complete systems rather than individual chips. Etched was founded in 2022 and is based in San Jose, California.

Company size
201-500
Company stage
Series D
Total funding
$1.6B
Headquarters
San Jose, CA
Founded
2022

Recently Posted Jobs

Explore more opportunities at Etched

See more jobs