Principal Machine Learning Engineer

Electronic Arts creates next-level entertainment experiences that inspire players and fans around the world. Here, everyone is part of the story. Part of a community that connects across the globe. A place where creativity thrives, new perspectives are invited, and ideas matter. A team where everyone makes play happen.

At EA, we believe games are powerful because they bring together multiple ways people engage: play, watch, create, and connect. And increasingly, the biggest entertainment platforms aren't just places to consume content—they're places where communities build.

Creator-made content is already a proven part of EA's history—from community creation tools in Battlefield to The Gallery in The Sims 4. We believe new creative technologies and tools will expand how players engage with and contribute to our experiences, supported by thoughtful product design, safety systems, and global reach. Our focus is on enabling more players to participate in creative expression by making creation easier, safer, and more rewarding.

As a Principal ML Engineer, Training Pipelines, you will own the systems that prepare and deliver multimodal game data for large-scale model training. You will set the architecture for these training input pipelines and improve data throughput, accelerator utilization, reproducibility, and recovery from failures.

Your responsibilities will initially focus on hands-on individual contribution and technical leadership, with potential to take on people-management responsibilities as the team grows. You will partner with researchers and central data and compute teams to turn evolving model requirements into reusable training capabilities.

Responsibilities:

• You will architect and build shared training input pipelines from certified game capture to accelerators, covering storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across multiple training frameworks.

• You will define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, acceptance criteria, and reproducible splits.

• You will build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and corrupt data, including elastic restarts and changes to cluster size or parallelism.

• You will design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, titles, and capture conditions, accounting for worker/rank locality and distributed parallelism.

• You will benchmark and enforce throughput, data-wait time, step time, and accelerator utilization and stalls. Yu will model FLOPs utilization (MFU) where applicable, per-rank skew, scaling efficiency, restart time, reliability, and cost targets through thousands of accelerators.

• You will diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators, including stragglers, faulty hardware, corrupt samples, and imbalanced batches.

• You will integrate compilers, runtimes, optimized kernels, low-precision training, and hardware-accelerated media processing. You will maintain efficient data delivery and recovery under FSDP/HSDP, tensor, context, and pipeline parallelism across heterogeneous accelerator environments.

• You will lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads. You will translate model and data-access requirements into scalable systems building on shared infrastructure services, while researchers retain model-research ownership.

Qualifications:

• 20+ years of development experience, including owning production systems for distributed-training input, with measurable improvements in throughput, accelerator utilization, step time, reliability, recovery time, or cost

• Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for single jobs spanning thousands of accelerators

• Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and optimization on one or more accelerator platforms such as GPUs, TPUs, or AWS Trainium

• Pre-training experience and an understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency

• Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.

• Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, and elastic restart; experience with schemas, dataset versioning, lineage, quality, and reproducible training/evaluation

• Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults

Pay Transparency - North America

* British Columbia (depending on location e.g. Vancouver vs. Victoria) *$204,600 - $287,500 CAD

* California (depending on location e.g. Los Angeles vs. San Francisco) *$227,300 - $344,700 USD