Tesla

Makes electric cars, robotaxis and grid batteries

Software Engineer - SI Workload Scheduling, SI Infra Supercomputing

Platform EngineeringUpdated on Nov 10

Full-timePalo Alto, CA, USANo salary listedIn-person

About the Job

Requirements

Deep hands-on experience with Slurm (configuration, plugins, scheduling algorithms, and accounting) or comparable cluster/workload scheduling systems such as PBS Pro/OpenPBS, LSF, Grid Engine, HTCondor, or Kubernetes-based batch schedulers.
Strong knowledge of Python and Linux systems administration.
Experience operating and tuning HPC job schedulers at a scale of thousands of nodes or more.
Understanding of GPU cluster topologies, networking fabrics, and how scheduling decisions impact training performance.
Experience building tooling and automation for cluster operations, including job orchestration, resource monitoring, and capacity planning.
Familiarity with containerization and orchestration technologies such as Docker, Kubernetes, and Pyxis/Enroot.
Understanding of modern machine learning training workflows and resource requirements.
Experience with parallel programming concepts and distributed systems.
Ability to adapt to dynamic requirements of SI software engineering and contribute across the SI and HPC software stack.

Responsibilities

Own and evolve the Slurm scheduler configuration, plugins, and policies that govern job scheduling across GPU supercomputers.
Design and implement scheduling policies that maximize cluster utilization, minimize job queue times, and enforce fair-share allocation across teams and priorities.
Build tooling and automation around job submission, preemption, backfill, and resource reservation to support the dynamic needs of SI engineering.
Develop monitoring and observability infrastructure that provides real-time visibility into cluster utilization, job throughput, and scheduling efficiency.
Debug and identify the root causes of scheduling failures and resource contention issues across thousands of nodes, and implement fixes to prevent recurrence.
Coordinate with the operations team managing the training cluster to maintain high availability and job throughput.
Work closely with the ML team to understand workload patterns and evolving resource requirements.

Category	Engineering & Information Technology
Location	Palo Alto, California
Requisition ID	269362
Type	Full-time

Job Description

As a Software Engineer within the Supercomputing SI Infrastructure team, you will work on scaling and optimizing our training compute clusters at the core of Robotaxi and the Optimus development. Tesla Supercomputing maintains, on premise, some of the largest supercomputers in the world. We’re currently scaling 100K+ GPU clusters, which are central to developing our autonomy capabilities. Robustly managing such a large system of Super Intelligence training and HPC clusters requires software design through the stack: from the operating system to workload scheduling to ultimately the training loop. We are building and improving the workload scheduling systems that govern how hundreds of thousands of GPUs are allocated across training jobs, experiments, and data pipelines. In this role, you will own the Slurm-based scheduler at the heart of our SI HPC compute infrastructure, ensuring that cluster resources are utilized efficiently and that engineers can iterate quickly without friction.

Job Responsibilities

Own and evolve the Slurm scheduler configuration, plugins, and policies that govern job scheduling across our GPU supercomputers
Design and implement scheduling policies that maximize cluster utilization, minimize job queue times, and enforce fair-share allocation across teams and priorities
Build tooling and automation around job submission, preemption, backfill, and resource reservation to support the dynamic needs of SI engineering
Develop monitoring and observability infrastructure to provide real-time visibility into cluster utilization, job throughput, and scheduling efficiency
Debug and root cause scheduling failures and resource contention issues across thousands of nodes and implement fixes to prevent recurrence
Coordinate with the operations team managing the training cluster to maintain high availability and job throughput
Work closely with the ML team to understand workload patterns and evolving resource requirements

Job Requirements

Members of the Supercomputing SI Infrastructure team are expected to be adaptable to the dynamic requirements of SI software engineering and capable of contributing across all parts of the SI and HPC software stack
Deep hands-on experience with Slurm (configuration, plugins, scheduling algorithms, accounting) or comparable cluster/workload scheduling systems (e.g., PBS Pro/OpenPBS, LSF, Grid Engine, HTCondor, Kubernetes-based batch schedulers)
Strong knowledge of Python and Linux systems administration
Experience operating and tuning HPC job schedulers at scale (>=thousands of nodes)
Understanding of GPU cluster topologies, networking fabrics, and how scheduling decisions impact training performance
Experience building tooling and automation for cluster operations (job orchestration, resource monitoring, capacity planning)
Familiarity with containerization and orchestration technologies (Docker, Kubernetes, Pyxis/Enroot)
Understanding of modern machine learning training workflows and resource requirements
Experience with parallel programming concepts and distributed systems

About the Company

Tesla

Makes electric cars, robotaxis and grid batteries

Tesla makes electric cars and batteries that store power for homes and utilities, and it develops self-driving software and humanoid robots. In 2023, its Model Y became the first electric vehicle to rank as the world's best-selling car. Buyers order the Model 3 sedan, Model Y SUV, or Cybertruck pickup directly from Tesla instead of through dealers, and they charge on its Supercharger fast-charging network. It also sells Megapack and Powerwall batteries, runs a robotaxi service, and charges drivers a monthly fee for self-driving software that still needs their supervision. Tesla was founded in 2003, with Elon Musk as its lead early investor and a recognized co-founder, and it is based in Austin, Texas.

Company size: 10,001+
Company stage: IPO
Headquarters: Austin, TX
Founded: 2003

Recently Posted Jobs

Explore more opportunities at Tesla

See more jobs