L

Lambda

Cloud-based GPU services for AI training

Senior HPC Engineer - Fleet Engineering

Site Reliability EngineeringConfirmed live in the last 24 hours
Full-time

About the Job

Requirements

  • Have 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role.
  • Have a strong understanding of modern AI infrastructure, from GPU architectures to hardware performance optimization.
  • Have a strong understanding of Linux-based systems in a distributed environment.
  • Be experienced configuring and troubleshooting InfiniBand, RoCE, CLOS fabrics, 100GbE, Ethernet and switching, GPU-direct, and NCCL environments.
  • Have a solid understanding of Python and Go, with experience working with software engineering teams to improve internal tooling.
  • Have experience with monitoring and alerting tools such as Prometheus, Grafana, and ClickHouse.
  • Have proficiency in automation and configuration management tools such as Ansible and Terraform.
  • Have problem-solving and troubleshooting skills and attention to detail.

Responsibilities

  • Build and operate monitoring and alerting for cluster health, including fabric, GPU, power and thermal, and job-level signals, to detect and respond to issues proactively.
  • Remotely deploy and configure large-scale HPC clusters for AI workloads using automation wherever possible.
  • Automate cluster lifecycle management for operating systems, firmware, drivers, and networking using code such as Ansible and Terraform rather than manual processes.
  • Create runbooks and automated remediations for common cluster failure modes so Support and HPC Support can run them safely.
  • Troubleshoot and resolve cluster issues across InfiniBand and RoCE, NCCL, GPU-direct, fabric, switching, and power, working closely with on-site deployment teams.
  • Participate in on-call rotations and lead incident response for cluster-level problems.
  • Contribute to and maintain Standard Operating Procedures, and provide clear requirements to other engineering teams on simplification, stability, and operational efficiency.

Desired Qualifications

  • Have experience with machine learning or deep learning frameworks such as PyTorch and TensorFlow, and benchmarking tools such as DeepSpeed and MLPerf.
  • Have knowledge of containerization and orchestration technologies such as Docker and Kubernetes.
  • Have experience building or operating HPC resources.
  • Have depth in the NVIDIA hardware and firmware ecosystem.
  • Have experience with data center power and thermal design.
  • Have a background in chaos engineering or similar reliability testing methodologies.
  • Understand compliance frameworks such as SOC 2 and ISO 27001.

About the Company

Lambda

Cloud-based GPU services for AI training

Lambda Labs provides cloud-based GPU services for AI training and inference via its AI Developer Cloud, powered by NVIDIA GH200 Grace Hopper hardware to efficiently train large language models and generative AI. Customers access on-demand or reserved GPUs and are billed by the hour (for example, NVIDIA H100 at about $1.99/hour); it also offers Lambda Echelon, where clients own infrastructure but pay a monthly hosting and support fee, and Lambda Stack, a software suite that helps ML teams install and upgrade tools like PyTorch, TensorFlow, CUDA, cuDNN, and NVIDIA drivers. The company differentiates itself through competitive pricing, available high-end hardware, and an integrated ML software stack that streamlines workflows. Its goal is to enable scalable, affordable AI development and deployment by providing flexible GPU access, reliable hosting, and easy software deployment for teams working with large models.

Company size
501-1,000
Company stage
Late Stage VC
Total funding
$9.1B
Headquarters
San Jose, CA
Founded
2012

Recently Posted Jobs

Explore more opportunities at Lambda

See more jobs