M

Microsoft

Builds workplace software, PCs and cloud services

Software Engineer - AI Infra and Model Foundry

Applied Machine LearningConfirmed live in the last 24 hours
Full-timeRedmond, WA, USA$142.8k - $304.2k /yrIn-person

About the Job

Requirements

  • A Bachelor's Degree in Computer Science or a related technical field and 6 or more years of technical engineering experience coding in languages including C, C++, C#, Java, JavaScript, Python, Rust, or Go, or equivalent experience.
  • Experience designing, building, and operating distributed systems or high-scale production infrastructure, including machine learning infrastructure, model serving, distributed compute, cloud platforms, or another performance-critical systems domain.
  • Demonstrated experience leading technically ambitious, multi-team infrastructure initiatives from architecture through dependable production operation and measurable impact.
  • Experience influencing technical direction across organizational boundaries and mentoring engineers, supported by strong systems fundamentals in concurrency, networking, storage, resource management, reliability, or performance analysis.
  • Experience serving large language models or other large generative models in production.
  • Experience with containers, schedulers, orchestration systems, and large-scale cloud infrastructure.
  • Experience building training, post-training, evaluation, or experiment-management platforms.
  • Familiarity with model-serving techniques such as continuous batching, prefix or KV caching, disaggregated prefill and decode, speculative decoding, parallelism, quantization, and model compilation.
  • Experience operating highly available, GPU-backed services with demanding latency, throughput, capacity, and cost requirements.
  • Experience making hardware-software co-design decisions and improving accelerator utilization or fleet economics at substantial scale.
  • Demonstrated written and verbal communication skills, including communicating system tradeoffs and operational risks to cross-functional partners.

Responsibilities

  • Set technical direction and lead multi-quarter initiatives for scalable frontier-model infrastructure, from experimentation and training through evaluation, deployment, and inference.
  • Develop high-throughput, low-latency model-serving systems that efficiently use GPUs and other accelerators.
  • Build reusable platform capabilities for model onboarding, versioning, packaging, rollout, routing, autoscaling, monitoring, and rollback.
  • Improve inference performance and fleet efficiency through batching, prefix and KV caching, workload placement, scheduling, parallelism, quantization, compilation, and memory optimization.
  • Create reliable interfaces and developer workflows that help research and product teams move models from experiments to production quickly and safely.
  • Define service-level objectives and build observability, benchmarking, capacity-management, and regression-detection systems for model quality, latency, throughput, reliability, and cost.
  • Diagnose and resolve bottlenecks across distributed systems, networking, storage, runtimes, model architectures, and accelerator hardware.
  • Lead the response to complex production incidents, drive root-cause analysis, and turn operational lessons into durable platform improvements.
  • Partner with researchers and reinforcement learning engineers to support large-scale experiment, rollout, evaluation, and data-generation workloads.
  • Establish technical and operational guardrails that allow teams to move quickly without creating regressions in availability, security, privacy, performance, or fleet economics.
  • Mentor engineers, raise the quality of architecture and design reviews, and contribute to an engineering culture grounded in knowledge sharing and technical excellence.

About the Company

Microsoft

Builds workplace software, PCs and cloud services

Microsoft builds workplace software, operating systems and cloud infrastructure for consumers, businesses and government agencies, from small organizations to global enterprises. Microsoft 365 combines Word and Excel with Teams communication and Copilot AI features, while Windows runs PCs and Surface supplies the company's own computers. Azure rents computing capacity, storage and databases to build and run web applications. Microsoft also sells Xbox consoles and games, collecting revenue through software licenses, subscriptions, usage charges and product sales. Founded in 1975 by Bill Gates and Paul Allen, Microsoft is headquartered in Redmond, Washington.

Company size
10,001+
Company stage
IPO
Headquarters
Redmond, WA
Founded
1975

Recently Posted Jobs

Explore more opportunities at Microsoft

See more jobs