Platform Engineer - Self Hosting Llms

The role

You will build the LLM access layer for agents on the PhysicsX platform. That layer serves self-hosted open models and routes each task to the model, or combination of models, that best fits its constraints.

We want to bring deep, production experience in self-hosted LLM serving and model routing into PhysicsX. You will establish the capability to run open models reliably and choose between self-hosted and external models according to the needs of each workload. You will set the technical direction for inference engines such as vLLM and SGLang, and serving platforms such as KServe. You will build foundations that product, delivery and customer teams can rely on.

You will own both. You will learn our requirements and define the implementation, rather than inherit a prescribed gateway design. You will wire serving and routing into the existing platform, identity and observability systems.

In the near term, you’ll make the platform’s agentic capabilities work out of the box, with no customer-supplied API key needed. Longer term, you’ll build the serving layer into a flexible routing foundation that balances competing priorities, such as cost, quality and data sensitivity, as our needs change.

What you will do

• Make self-hosted serving and model routing work in air-gapped deployments, including sites where the hardware, connectivity and operational setup force constrained solutions.
• Own LLM packaging, registration, deployment, health checks, upgrades, rollback and retirement for supported LLMs.
• Evaluate and integrate serving platforms such as KServe, including inference services, serving runtimes, model loading and rollout controls.
• Define, model, and implement service levels for availability, quality, and performance metrics (e.g. time to first token, inter-token latency, and tail latency).
• Measure and improve throughput, GPU utilisation, memory use and cost per token under representative agent workloads.
• Tune continuous and chunked batching, KV and prefix caching, quantisation, speculative decoding and tensor, data or expert parallelism where appropriate.
• Evaluate advanced techniques such as cache-aware routing and prefill/decode disaggregation when workload evidence justifies their complexity.
• Build autoscaling, admission control and capacity management around GPU and serving signals rather than generic CPU metrics.
• Make every routing decision observable and explainable, and use evaluation evidence before adding specialist routes.
• Work with Research and Post-Training specialists to move candidate and tuned models through evaluation into supported production routes.
• Help determine how this capability should grow and, over time, shape the team around it.

What you bring

Where you need to be strongest

• You have operated inference engines such as vLLM or SGLang, or serving platforms such as KServe, under production traffic, latency and reliability constraints.
• You can explain the boundary between serving orchestration and inference execution, including how deployment, scaling and runtime configuration affect performance.
• You have built or operated model routing across more than one provider, and can explain how you balanced cost, quality, latency, failure and policy.
• You can describe the serving architecture you operated, the bottlenecks you found, the changes you made and the measured effect on latency, throughput, reliability or cost.
• You understand LLM inference as a systems problem: batching, scheduling, KV-cache behaviour, memory pressure, model parallelism and GPU utilisation.
• You have built and operated GPU workloads on Kubernetes, including deployment, scaling, observability and failure recovery.
• You can turn fast-moving open-source engines and provider APIs into a supportable product with versioning, compatibility testing, rollback and clear operational boundaries.
• You are a strong systems engineer who remains hands-on in architecture, implementation, profiling and production debugging.
• Your experience goes materially beyond local model runners or demonstrations; tools such as Ollama are not a substitute for production serving ownership.

Foundations

• Strong Python plus a systems language such as Go, Rust or C++.
• Linux, containers, Kubernetes, networking, storage and distributed-systems fundamentals.
• GPU observability and performance analysis across engine, system and accelerator metrics.
• Model formats, quantisation, tokenisation and OpenAI-compatible serving APIs.
• Traffic management, streaming, overload protection, health checking and failure recovery.
• Integration with existing identity, authorisation, secrets and workload-isolation systems.
• Clear technical writing, design review, incident response and cross-team communication.

Nice to have

• Contributions to KServe, vLLM, SGLang or adjacent inference, gateway or orchestration projects.
• Experience modifying engine internals, profiling kernels or diagnosing CUDA and collective-communication issues.
• Experience with mixture-of-experts models, expert parallelism, speculative decoding or disaggregated serving.
• Experience routing by data-sensitivity policy, evaluation signals or multi-model review workflows.
• Experience with model serving in customer-operated, regulated or fully disconnected environments.
• Familiarity with Kata Containers, confidential computing or strong multi-workload isolation.
• Experience integrating serving with post-training rollouts and evaluation workloads.