Machine Learning Infrastructure Engineer

We're Zeromatter. Our mission is to accelerate how robotics and autonomy is built, tested, and trained by leveraging simulation to the fullest extent possible. Part of what we're building to accomplish this is a cloud-based sensor simulation platform that is beyond the state of the art.

We are looking for an individual who can support us in accomplishing this mission. You should apply if you're interested in working at a fast-moving startup where you can live up to your potential, unencumbered by office politics or team boundaries. Working at Zeromatter gives you an opportunity to work on technology that will soon be used by various well-known companies who have strong simulation needs.

At Zeromatter, we believe in evaluating candidates based on their skills, experience, and potential. We are committed to providing equal opportunities to all applicants and believe that having a diverse team spanning a wide variety of backgrounds, academically as well as in general, is a key asset. If you're passionate about Physical AI, robotics and simulation and eager to make a difference in the field, we'd love to hear from you!

Role description

• Design, deploy, and run our GPU compute across cloud providers and on-prem: provisioning, job scheduling (Kubernetes, Slurm, or Ray), autoscaling, quotas, and cost controls

• Build distributed training infrastructure for perception, world models, and RL policies, including multi-node PyTorch DDP/FSDP training, checkpointing, and fault recovery

• Make simulation and training run together without bottlenecks: high-throughput synthetic sensor data generation, plus RL and closed-loop training with the simulator in the loop

• Build the data platform for petabyte-scale simulated and real multimodal data: shard formats, caching tiers, dataset versioning, and lineage, so jobs are never input-bound

• Ship evaluation infrastructure that runs models in closed loop across thousands of simulated scenarios and reports metrics engineers trust

• Stand up inference and serving for models inside the platform, optimized for real-time simulation and, for some customers, edge hardware

• Own observability: per-job GPU utilization, step time, input wait, and failure classification, so every GPU-hour is accounted for

• Help customers run Zeromatter-based training in their own cloud or on-prem environments, and set engineering standards as an early member of the ML infra team

Responsibilities

• Ideally, you have an engineering degree in a related field (e.g. computer science, mechanical engineering, physics, math, robotics, etc.)

• 5+ years of software engineering, with substantial production work on distributed systems or ML infrastructure

• Hands-on experience building and operating GPU clusters for training at meaningful scale (roughly 100+ GPUs), including failures that only appear at scale: hung collectives, stragglers, input-bound jobs, stranded capacity

• Strong Python, plus one systems language such as C++, Rust, or Go

• Working knowledge of PyTorch and distributed training concepts (DDP, FSDP, NCCL, sharding, checkpointing)

• Experience with Kubernetes and at least one major cloud (AWS, GCP, or Azure) or large on-prem GPU infrastructure

• Comfort with infrastructure as code (Terraform or equivalent), containers, and CI/CD

• Sound judgment on trade-offs such as cloud versus on-prem, build versus buy, and speed versus cost, and clear communication with the engineers who use your platform

Qualifications

• Ideally, you have an engineering degree in a related field (e.g. computer science, mechanical engineering, physics, math, robotics, etc.)

• ML infra experience in autonomy, robotics, aerospace, or another simulation-heavy domain

• RL or closed-loop training infrastructure, where a simulator or environment runs inside the training loop

• Familiarity with GPU rendering and graphics APIs (Vulkan, CUDA interop), or feeding rendered sensor data into training

• Large-model training: tensor and pipeline parallelism, NCCL tuning, mixed precision

• GPU profiling and telemetry tools such as Nsight Systems, the PyTorch profiler, and DCGM

• Training-native data formats (WebDataset, Parquet, Lance) and GPU-side video or image decode

• Ray or KubeRay, Kueue or Volcano, and experiment tracking (Weights & Biases, MLflow)

• HPC background: schedulers, parallel file systems, and high-speed interconnects (InfiniBand, RoCE)

• Deploying and optimizing models for edge hardware (TensorRT, ONNX), including in customer, air-gapped, or regulated environments