Bloomberg

Global financial data, news, analytics provider

Senior Software Engineer - AI Inference

Applied Machine LearningConfirmed live in the last 24 hours

Full-timeNew York, NY, USAIn-person

About the Job

Requirements

At least 5 years of professional software engineering experience.
Experience designing, building, and operating production distributed systems.
Strong systems intuition and a track record of debugging and optimizing performance-critical services.
Ability to own problems end-to-end and quickly ramp up in unfamiliar technical areas.
At least 4 years of demonstrated experience working with an object-oriented programming language.
A degree in Computer Science, Electrical Engineering, or equivalent practical experience.

Responsibilities

Design and build scalable infrastructure for both online and offline inference workloads.
Lead integration of high-performance inference runtimes and serving frameworks, including TensorRT, vLLM, ONNX, and Triton.
Drive architecture and technical decisions across Bloomberg’s inference platform, balancing latency, throughput, reliability, and cost.
Partner across engineering teams to improve model deployment, observability, and production performance.
Mentor junior engineers on system design, debugging, and performance optimization.
Autoscale a heterogeneous compute fleet to match supply and demand across diverse inference workloads.
Build production-grade deployment pipelines to safely roll out new models to millions of users.
Develop new inference capabilities such as structured sampling, prompt caching, and advanced serving optimizations.
Analyze observability data from real production workloads to improve latency, throughput, and resource efficiency.

Desired Qualifications

Experience deploying and operating machine learning systems at scale.
Experience with inference optimization techniques such as batching, caching, request scheduling, or memory-aware serving.
Familiarity with PyTorch and GPU software stacks such as CUDA and NCCL.
Exposure to high-performance interconnects and distributed computing technologies such as NVLink, InfiniBand, or MPI.
Experience with Kubernetes and cloud-native infrastructure.
Experience with load balancing, request routing, or traffic management systems.

Our team:

Join the team that is building the core infrastructure for AI at Bloomberg. The Bloomberg AI Inference Platform provides production-grade managed infrastructure for hosting, deploying, and serving all machine learning models, both predictive and cutting-edge generative models. We abstract away infrastructure complexity, empowering engineering teams to focus on creating intelligent applications with guaranteed scalability, performance, and governance. Our platform is built on the open-source KServe project, and the CNCS AI Inference team is a primary contributor to its development.

We'll trust you to:

Design and build scalable infrastructure for both online and offline inference workloads.
Lead integration of high-performance inference runtimes and serving frameworks, including TensorRT, vLLM, ONNX, and Triton.
Drive architecture and technical decisions across Bloomberg’s inference platform, balancing latency, throughput, reliability, and cost.
Partner across engineering teams to improve model deployment, observability, and production performance.
Mentor junior engineers on system design, debugging, and performance optimization.

You'll need to have:

5+ years of professional software engineering experience.
Experience designing, building, and operating production distributed systems.
Strong systems intuition and a track record of debugging and optimizing performance-critical services.
Ability to own problems end-to-end and quickly ramp up in unfamiliar technical areas.
4+ years of demonstrated experience working with an object-oriented programming language.
A degree in Computer Science, Electrical Engineering, or equivalent practical experience.

We'd love to see:

Experience deploying and operating machine learning systems at scale.
Experience with inference optimization techniques such as batching, caching, request scheduling, or memory-aware serving.
Familiarity with PyTorch and GPU software stacks such as CUDA and NCCL.
Exposure to high-performance interconnects and distributed computing technologies such as NVLink, InfiniBand, or MPI.
Experience with Kubernetes and cloud-native infrastructure.
Experience with load balancing, request routing, or traffic management systems.

Representative projects:

Autoscaling a heterogeneous compute fleet to match supply and demand aross diverse inference workloads.
Building production-grade deployment pipelines to safely roll out new models to millions of users.
Developing new inference capabilities such as structured sampling, prompt caching, and advanced serving optimizations.
Analyzing observability data from real production workloads to improve latency, throughput, and resource efficiency.

Salary Range = 160,000 - 240,000 USD Annual + Benefits + Bonus

The referenced salary range is based on the Company's good faith belief at the time of posting. Actual compensation may vary based on factors such as geographic location, work experience, market conditions, education/training and skill level.

We offer one of the most comprehensive and generous benefits plans available and offer a range of total rewards that may include merit increases, incentive compensation (exempt roles only), paid holidays, paid time off, medical, dental, vision, short and long term disability benefits, 401(k) +match, life insurance, and various wellness programs, among others. The Company does not provide benefits directly to contingent workers/contractors and interns.

Discover what makes Bloomberg unique - watch our podcast series for an inside look at our culture, values, and the people behind our success.

About the Company

Bloomberg

Global financial data, news, analytics provider

Bloomberg provides financial data, news, and analysis to help professionals make informed decisions and operate more transparently and efficiently. Its products deliver trusted information and insights through technology platforms that connect global financial communities, enable collaboration, and support decision-making. The company differentiates itself by offering broad, reliable data and news across markets, strong customer focus, and a culture that emphasizes collaboration, broad perspective, and doing the right thing for clients, people, and communities. Bloomberg’s goal is to improve market transparency and efficiency by delivering accurate information and tools that empower customers to act confidently and ethically.

Company size: 10,001+
Company stage: Not listed
Total funding: Not listed
Headquarters: New York City, New York
Founded: 1981

Recently Posted Jobs

Explore more opportunities at Bloomberg

See more jobs