Tesla

Makes electric cars, robotaxis and grid batteries

Software Engineer - Super Intelligence Infrastructure

Platform EngineeringConfirmed live in the last 24 hours

Full-timePalo Alto, CA, USANo salary listedIn-person

About the Job

Requirements

Apply strong software engineering practices to infrastructure, writing maintainable automation and services rather than only operating tickets.
Be proficient in Python and/or Bash; additional programming languages are a plus.
Have deep production Linux experience, including boot or image management, configuration management, networking, performance debugging, and package or driver realities on physical hardware.
Have experience with GPU cluster operations or super-intelligence training infrastructure, including familiarity with the NVIDIA stack, drivers, CUDA, and multi-node job behavior.
Know HPC schedulers and workload managers such as Slurm or LSF; Kubernetes exposure for machine-learning or infrastructure workloads is a plus.
Have experience with high-performance networking, including RDMA, RoCEv2, InfiniBand, congestion behavior, and host-side tuning for collective-heavy training.
Have experience with parallel or high-performance storage such as Lustre, GPFS/Spectrum Scale, BeeGFS, or cloud/parallel equivalents, including failure modes under training I/O.
Have experience with configuration management or infrastructure-as-code-style automation at fleet scale, such as Ansible or an equivalent.
Be willing to participate in a 24×7 on-call rotation for critical super-intelligence training infrastructure.
Have a degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.

Responsibilities

Keep the training fleet reliable and efficient so research and production training are not blocked by infrastructure toil or silent degradation.
Replace manual operations with reliable, testable software, APIs, job pipelines, operators/controllers, and libraries that other engineers and on-call staff can depend on.
Build metrics, logging, alerting, and automated remediation, including Prometheus/Grafana-based stacks, to detect unhealthy nodes, GPUs, fabric or storage faults, and scheduler failures, and safely fence or repair them at scale.
Model failure domains, reduce the impact of hardware and software faults on long-running multi-node jobs, and improve effective capacity by reducing idle accelerators and silent stragglers.
Build and extend performance-oriented tools that measure and improve throughput and stability across operating-system, network, storage, and runtime layers for distributed training workloads.
Engineer against and improve HPC/super-intelligence scheduling and orchestration surfaces, including Slurm/LSF and related platforms, plus containers/Kubernetes where used, so capacity is allocatable, debuggable, and efficient.
Partner with machine-learning infrastructure users, networking, data center, and hardware/vendor teams to bring up capacity and provide clean software interfaces rather than relying only on ad hoc runbooks.
Participate in a 24×7 rotation for critical training infrastructure, invest in service-level indicators/objectives and runbooks as code, and implement permanent fixes that reduce pages over time.

Desired Qualifications

Additional programming languages are a plus.
Kubernetes exposure for machine-learning or infrastructure workloads is a plus.

Category	Engineering & Information Technology
Location	Palo Alto, California
Requisition ID	252825
Type	Full-time

Job Description

Tesla’s Super Intelligence Infrastructure team builds the software that turns GPUs and custom silicon into reliable, efficient training capacity for Autopilot, Optimus, Robotaxi, and Dojo. We own the systems layer between bare metal and researchers: cluster lifecycle, fleet automation, observability, scheduling interfaces, and the control-plane software that keeps multi-node training jobs healthy at scale.<p><br>In this role, you'll work as a software engineer in a high-performance environment, shipping production services and tooling that operate thousands of accelerated machines, not only configuring them by hand. You'll design, build, and operate software that provisions, monitors, heals, and optimizes our SI training fleets. You treat the cluster as a product: APIs, automation pipelines, reliability signals, and abstractions that make supercomputing capacity safe and fast for ML users. You’ll focus on distributed-systems and systems-software problems (hosts, accelerators, fabric, storage, schedulers), then encode fixes into software so the same failure class doesn’t require a hero page again. Scope and level scale with impact, from owning major automation surfaces to leading design across fleet-wide platforms.</p>

Job Responsibilities

Keep the training fleet up; Drive reliability and efficiency of GPU clusters so research and production training progress is not gated on infrastructure toil or silent degradation
Ship automation as product; Replace manual ops with reliable, testable software, APIs, job pipelines, operators/controllers, and libraries other engineers and on-call can depend on
Own observability and remediation paths; Build metrics, logging, alerting, and automated remediation (e.g. Prometheus/Grafana-based stacks) that detect bad nodes, sick GPUs, fabric/storage faults, and scheduler failures and fence or repair safely at scale
Raise training reliability through software; Model failure domains, reduce blast radius of HW/SW faults on long-running multi-node jobs, and improve effective capacity (less idle accelerators, fewer silent stragglers)
Develop performance-oriented systems tooling; Build and extend tools that measure and improve throughput/stability across OS, network, storage, and runtime layers for real distributed training workloads
Integrate with scheduling and workload platforms; Engineer against and improve HPC/SI scheduling and orchestration surfaces (e.g. Slurm/LSF and related platforms; containers/K8s where used) so capacity is allocatable, debuggable, and efficient
Partner across the stack working with ML infrastructure users, networking, data center, and hardware/vendor partners to bring up capacity and expose clean software interfaces, not ad-hoc runbooks alone
Participate in a 24×7 rotation for critical training infra; invest in SLIs/SLOs, runbooks-as-code, and permanent fixes that drive pages down over time

Job Requirements

Strong software engineering instincts applied to infrastructure: you write maintainable automation and services, not only operate tickets
Proficiency in Python and/or Bash (additional languages a plus); comfort building tools others will run in production
Deep Linux experience in production (boot/image or config management, networking stack, performance debugging, package/driver realities on real hardware)
GPU cluster operations or SI training infrastructure (NVIDIA stack, driver/CUDA ecosystem familiarity, multi-node job behavior)
HPC schedulers and workload managers (Slurm, LSF, or similar); exposure to Kubernetes for ML/infra workloads a plus
High-performance networking: RDMA, RoCEv2, InfiniBand, congestion behavior, and host-side tuning for collective-heavy training
Parallel / high-performance storage (Lustre, GPFS/Spectrum Scale, BeeGFS, or cloud/parallel equivalents) and its failure modes under training I/O
Experience with configuration management / IaC-style automation at fleet scale (e.g. Ansible, or equivalent)
Willingness to participate in a 24×7 on-call rotation for critical SI training infrastructure (cadence shared during hiring; escalation and automation are first-class goals of the role)
Degree in CS, EE, or related field, or equivalent practical experience

About the Company

Tesla

Makes electric cars, robotaxis and grid batteries

Tesla makes electric cars and batteries that store power for homes and utilities, and it develops self-driving software and humanoid robots. In 2023, its Model Y became the first electric vehicle to rank as the world's best-selling car. Buyers order the Model 3 sedan, Model Y SUV, or Cybertruck pickup directly from Tesla instead of through dealers, and they charge on its Supercharger fast-charging network. It also sells Megapack and Powerwall batteries, runs a robotaxi service, and charges drivers a monthly fee for self-driving software that still needs their supervision. Tesla was founded in 2003, with Elon Musk as its lead early investor and a recognized co-founder, and it is based in Austin, Texas.

Company size: 10,001+
Company stage: IPO
Headquarters: Austin, TX
Founded: 2003

Recently Posted Jobs

Explore more opportunities at Tesla

See more jobs