E

Etched

Designs chips and servers for running AI models

Infrastructure Engineer - Labs

Platform EngineeringPosted on Nov 12
Full-timeSan Jose, CA, USA

About the Job

Requirements

  • Have at least 4 years of hands-on infrastructure, data center systems, or bare-metal operations experience in on-premises environments, including physical hardware work.
  • Be able to work across servers, power circuits and PDUs, Linux, networking, and hardware, and troubleshoot failures in those areas.
  • Have strong bare-metal Linux systems skills, including kernel and driver debugging, PCIe and NIC troubleshooting, storage, and boot issues.
  • Have provisioned bare-metal fleets over the network using PXE/iPXE, DHCP, and DNS, and used BMCs such as Redfish or IPMI for out-of-band management.
  • Write infrastructure automation using Ansible, YAML, Jinja2, and Python, and work in Git with pull requests, code review, and tests.
  • Have working knowledge of L2/L3 networking fundamentals, VLANs, LACP, switch configuration, and optics and copper cabling at 100G and above.
  • Have built or operated observability stacks such as Prometheus, Grafana, or VictoriaMetrics.
  • Have handled incident response and deployed software and firmware updates across a fleet without disrupting operations.
  • Be able to rack and unrack heavy servers using a lift, run cabling, and work hands-on in an active lab.
  • Be able to work on-site in San Jose and travel to Bay Area lab and data center sites.

Responsibilities

  • Own rack and server infrastructure across labs and data center sites, from handoff by forward-deployed engineers through bring-up, validation, and testing.
  • Deliver full-stack infrastructure for new lab and data center sites, including compute, networking, storage, DHCP, DNS, zero-touch provisioning, and NFS services.
  • Build and maintain bare-metal provisioning for servers, development kits, rack management controllers, and switches using PXE/iPXE, DHCP, BMC/Redfish-driven imaging, and golden images.
  • Troubleshoot Linux hosts on custom and early-stage hardware, including kernel, PCIe, NIC driver and firmware, bonding, storage, boot, and power issues; identify root causes and document them.
  • Connect racks to the network by configuring top-of-rack and uplink connections and planning VLANs and IP addresses, in partnership with network engineers.
  • Diagnose physical-layer and hardware faults, including link flaps, CRC errors, dirty or failed optics, power faults, and coolant leaks, and coordinate repairs and RMAs with lab staff and vendors.
  • Help manage the Slurm cluster used to reserve accelerator systems, including partitions, reservations, and maintenance drains, and expand storage and compute as programs grow.
  • Extend the Prometheus, VictoriaMetrics, and Grafana observability stack to every lab rack, with dashboards and alerts for health, power, and link status across sites.
  • Write infrastructure automation in Ansible and Python through reviewed and tested pull requests, and build tooling for controlled environments and individual lab teams.
  • Manage software releases and changes on lab systems under change control, respond to incidents, and keep runbooks, inventory, and design records current.
  • Stand up full infrastructure for the Milpitas hardware facility, including network, provisioning, storage, telemetry, and access, ahead of next-generation rack validation.
  • Build a portable, containerized provisioning stack using DHCP, PXE, HTTP, and ZTP to bring up servers and switches from the network.
  • Plan and build lab infrastructure for next-generation silicon bring-up so systems are reachable and imaged when engineering samples arrive.
  • Build a colocation cage from empty to operational by coordinating cross-connects and circuits, configuring top-of-rack switches and uplinks, and preparing racks for internal workloads.
  • Build a link- and hardware-health pipeline that detects rising bit error rates, CRC errors, and flapping links and identifies corrective actions.
  • Design an asset and inventory workflow linking serial numbers, MAC and IP addresses, and rack locations so repairs and RMAs are traceable.
  • Build high-throughput NVMe and NFS storage for loading model weights across lab racks, with documented layouts and benchmarks.
  • Move lab telemetry and firmware workflows to standard Redfish and OpenBMC interfaces as hardware platforms adopt them.

Desired Qualifications

  • Experience operating GPU, accelerator, or HPC clusters, especially liquid-cooled systems using in-rack CDUs, facility liquid cooling, leak detection, and automated shutdown.
  • Experience with OCP hardware, including Open Rack v3, DC bus bar power, and OpenBMC.
  • Experience in hardware bring-up and NPI environments (EVT, DVT, PVT) where platforms are still changing.
  • Experience with high-performance networking, including RoCE/RDMA fabrics, Tomahawk-class switches, Arista EOS or SONiC, and AEC, DAC, and optical cabling at 400G.
  • Experience with Slurm and Kubernetes cluster operations.
  • Storage engineering experience with NVMe, mdadm RAID, and NFS, including performance benchmarking.
  • Experience with asset management or DCIM systems.
  • Experience with colocation build-outs, including cross-connects, carrier circuits, and remote-hands coordination.
  • Experience with infrastructure identity and access tools such as FreeIPA, RADIUS, or TACACS+.
  • Experience with Terraform and operating RHEL and Ubuntu Linux distributions side by side.

About the Company

Etched

Designs chips and servers for running AI models

Etched designs AI chips and the server racks built around them for AI companies and cloud providers, focusing only on inference, the work of running trained models to answer requests. In 2024, it unveiled a chip hard-wired to run only transformers, the type of AI model behind ChatGPT, after betting at its founding that they would dominate AI. Etched develops its chips, racks, and software together, and each rack links many chips through a shared pool of fast memory so very large models with long inputs can answer quickly. It earns money by selling these complete systems rather than individual chips. Etched was founded in 2022 and is based in San Jose, California.

Company size
201-500
Company stage
Series D
Total funding
$1.6B
Headquarters
San Jose, CA
Founded
2022

Recently Posted Jobs

Explore more opportunities at Etched

See more jobs