Inference & Workload Optimization Lead

Inference & Workload Optimization Lead

Level: Principal (MTS 5) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role. Own how well customer AI workloads run on the capacity they have committed to, and build the autonomy that makes that expertise scale past one person’s attention.

This is not a role about serving models to our own agents; that already works. This is about making a client’s training and inference jobs measurably faster on their dedicated cluster, which makes the capacity they bought more valuable.

The problems you inherit

• Optimization is expert-dependent and manual. There is no systematic profiling of customer workloads and no repeatable playbook.

• No performance baselines. We cannot yet say what good looks like per hardware generation, which means we cannot distinguish a badly configured customer job from a well configured one.

• None of it is encoded. Whatever optimization knowledge exists lives in people’s heads rather than in agents that can apply it across every cluster.

• Internal model serving needs maturing, as a secondary duty. Serving works; routing is basic, with no cost or latency awareness.

• GPU capacity for internal inference is negotiated manually against a fleet largely committed to customers.

You will

• Profile customer training and inference workloads and find the real bottlenecks: data loading, collective communication, memory bandwidth, kernel efficiency, batching, parallelism layout

• Establish performance baselines and a definition of good per hardware generation, starting with current-generation NVIDIA hardware

• Turn that expertise into agent-executable optimization, so a fix reaches every cluster rather than the one you happened to look at

• Work with committed-capacity customers, directly or through the operating teams, on workload tuning

• Own internal model serving as a secondary responsibility, including cost and latency-aware routing and model version migration

• Advise on GPU allocation between internal inference and customer workloads

You have

• Deep GPU performance engineering: profiling, memory and bandwidth behaviour, and enough kernel-level understanding to know where time actually goes - Serving stack depth, for example vLLM, SGLang, or TensorRT-LLM, including batching, KV cache management, and quantization trade-offs - Distributed training and inference experience at scale: tensor, pipeline, and data parallelism, and collective communication tuning such as NCCL

• Fluency with profiling tooling, for example Nsight, DCGM, and framework-level profilers

• The ability to explain a performance finding to a customer and to encode it for an agent. These are different skills and this role needs both

Nice to have: CUDA kernel work, open-weight model deployment at scale, benchmark design, prior customer-facing technical experience