Staff Software Engineer

About the Role:

The ML Training Team’s core mandate is training models as fast as possible for the company. The team’s main focus is ensuring our models have 100% gpu utilization and can scale linearly from 8 gpus -> 256 gpus. We also invest in tooling to empower our MLEs, by building profiling/debugging tools, setting up efficiency monitoring and integrating our trainer into our experiment management system.

Responsibilities: 

• Setup efficiency monitoring for all our training jobs to identify models that need improvement
• Work with customer teams to benchmark/profile their jobs and make improvements
• Create standardized APIs for stack-wide abstractions like training datasets, bulk inference jobs, evaluation metrics
• Optimize dataloaders / training data formats to ensure high gpu utilization
• Optimize distributed training configurations (network topologies, sharding strategies, pipelines, etc).

Qualifications: 

• Experience: 5+ years as a SWE, ideally building infrastructure/customer facing product, experience in AV or robotics is also great.
•  Ideal Skills: 
  
  • Experience with both ML Platforms and building ML-based applications (bonus point if you have modeling experience).
  • Experience building scalable, reliable infra at a fast-paced environment.
  • Experience building or using ML infra built for a large number of customer teams.
  • A deep understanding of design tradeoffs and ability to articulate those tradeoffs and work with others on getting alignment.
  • Experience with building ML models or ML infra in the domains of autonomous vehicles, perception, and decision making (desirable but not required).
  • Experience with model training, model optimization, or large data processing pipelines.
  • Machine Learning Expertise is preferred but not necessary.
  • Knows how to push the GPU to its limit from Python to CUDA kernel level.
  • Built the inference or training loop for a large model (ideally with LLM flavor).
  • Shipped ML products (NLP, computer vision, recommender systems, etc.) at scale to make business impact.
  • Knows how to build low latency / high throughput batch or stream processing pipelines.
  • Knows how to write (readable) high performance C++.
  • Prior AV experience.
• Desired Attributes:
  • High customer empathy, able to communicate with customers well
  • Comfortable reading papers / keeping up with SOTA ML literature

#LI-AW1