Distributed Training Engineer
Liquid AI
Distributed Training Engineer
Liquid AI
Provides generative, edge-capable foundation models
Spun out of MIT CSAIL, we build general-purpose AI systems that run efficiently across deployment targets, from data center accelerators to on-device hardware, ensuring low latency, minimal memory usage, privacy, and reliability. We partner with enterprises across consumer electronics, automotive, life sciences, and financial services. We are scaling rapidly and need exceptional people to help us get there.
Our Training Infrastructure team is building the distributed systems that power our next-generation Liquid Foundation Models. As we scale, we need to design, implement, and optimize the infrastructure that enables large-scale training.
This is a high-ownership training systems role focused on runtime/performance/reliability (not a general platform/SRE role). You’ll work on a small team with fast feedback loops, building critical systems from the ground up rather than inheriting mature infrastructure.
We need someone who:
Loves distributed systems complexity: Our team builds systems that keeps long training runs stable, debugs training failures across GPU clusters, and improves performance.
Wants to build: We need builders who find satisfaction in robust, fast, reliable infrastructure.
Thrives in ambiguity: Our systems support model architectures that are still evolving. We make decisions with incomplete information and iterate quickly.
Aligns with team priorities and delivers: Our best engineers align with team priorities while pushing back with data when they see problems.
Design and build core systems that make large training runs fast and reliable
Build scalable distributed training infrastructure for GPU clusters
Implement and tune parallelism/sharding strategies for evolving architectures
Optimize distributed efficiency (topology-aware collectives, comm/compute overlap, straggler mitigation)
Build data loading systems that eliminate I/O bottlenecks for multimodal datasets
Develop checkpointing mechanisms balancing memory constraints with recovery needs
Create monitoring, profiling, and debugging tools for training stability and performance
Must-have:
Hands-on experience building distributed training infrastructure (PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP)
Experience diagnosing performance bottlenecks and failure modes (profiling, NCCL/collectives issues, hangs, OOMs, stragglers)
Understanding of hardware accelerators and networking topologies
Experience optimizing data pipelines for ML workloads
Nice-to-have:
MoE (Mixture of Experts) training experience
Large-scale distributed training (100+ GPUs)
Open-source contributions to training infrastructure projects
Training throughput has increased
Overall training efficiency/cost has improved
Training stability has improved (fewer failures, faster recovery)
Data loading bottlenecks are eliminated for multimodal workloads
Greenfield challenges: Build systems from scratch for novel architectures. High ownership from day one.
Compensation: Competitive base salary with equity in a unicorn-stage company
Health: We pay 100% of medical, dental, and vision premiums for employees and dependents
Financial: 401(k) matching up to 4% of base pay
Time Off: Unlimited PTO plus company-wide Refill Days throughout the year
Liquid AI builds and deploys foundation models based on liquid neural networks, focusing on efficient, on-device AI. It develops Liquid Foundation Models (LFMs), a family of generative AI models designed to be smaller and more computation-efficient than typical large language models, enabling deployment on edge devices with lower latency, better privacy, and reduced infrastructure costs. The approach includes end-to-end AI expertise and customizable architectures for enterprises that require real-time performance and private processing. Compared to traditional AI providers, Liquid AI emphasizes edge-ready, adaptable models that run efficiently on constrained hardware, and it targets enterprise-grade, private, reliable AI solutions. The company’s goal is to enable real-time, on-device AI at scale for businesses by offering compact, capable foundation models and the tools to customize them for specific applications.
Explore more opportunities at Liquid AI