Software Engineer 2 – AI Platform

What You'll Do:

Build the AI platform: APIs, SDKs, and self-service workflows so AI/ML engineers deploy, version, evaluate, and monitor workloads without touching raw infrastructure.

Own GPU infrastructure: Cluster design, provisioning, scheduling, isolation, and utilisation optimisation across shared multi-team demand.

Run inference in production: vLLM, Triton, KServe, or Ray — continuous batching, autoscaling, model routing, and multi-model serving against real latency and throughput SLOs.

Optimize relentlessly: Drive tokens/sec, p99 latency, GPU utilization, and cost per token through quantization, batching strategy, and capacity planning.

Engineer GPU-aware Kubernetes: GPU Operator, device plugins, GPU-aware scheduling, MIG partitioning, and distributed workloads over NCCL.

Debug the hard layer: GPU OOMs, driver and CUDA runtime mismatches, interconnect bottlenecks, throughput regressions.

Build the MLOps backbone: Model registry and versioning, CI/CD for AI workloads, and safe rollout/rollback.

Own the evaluation layer: Offline and online eval harnesses, golden datasets, LLM-as-a-judge scoring, accuracy and hallucination metrics, and automated regression gates so no model, prompt, or quantisation change ships without a measured quality verdict.

Make it observable: Response quality and accuracy drift, latency, throughput, GPU utilisation, and per-tenant cost — with SLOs, alerting, and incident response.

Secure and multi-tenant by default: AuthN/AuthZ, secrets, data protection, tenant isolation, and governed access to models and GPUs.

Codify and lead: Terraform for everything, HA and DR designed in, plus architecture direction and mentoring across teams.

Contribute beyond AI: Design and build secure, multi-tenant microservices and APIs on AWS, run thorough code reviews, and own feature delivery end-to-end with Product and Design.

What We're Looking For:

2-4 years building and operating production software and infrastructure, with senior ownership of systems end to end.

Bachelor's or Master's in Computer Science, Engineering, or equivalent practical experience.

Strong Python and/or Go — you write and review production services, not just scripts and manifests.

Deep Kubernetes and Docker: scheduling, resource management, operators, networking, debugging under load.

Strong Linux, networking, storage, and distributed systems fundamentals.

Production AWS/Azure/GCP experience — Lambda, API Gateway, EC2, S3, RDS and equivalents — with Terraform as your default way of working.

Working knowledge of SQL and NoSQL databases, including schema design and performance tuning.

Experience building and operating backend services and APIs in a multi-tenant SaaS product.

Leadership, code review, and mentoring skills, with end-to-end ownership of delivery in an agile environment.

Hands-on with a managed ML/AI platform — AWS SageMaker, Bedrock, GCP Vertex AI, or Azure ML — including where it fits and where self-managed infrastructure wins on cost or control.

Track record on HA, scalability, and DR in multi-tenant environments.

Solid observability and CI/CD practice — metrics, traces, SLOs, automated delivery.

Experience building or operating AI evaluation systems — accuracy and quality measurement, LLM-as-a-judge pipelines, eval datasets, and regression testing for models and prompts.

Background in platform engineering, infrastructure, SRE, distributed systems, AI infrastructure, or MLOps/LLMOps.

GPU & AI Experience (Required)

Hands-on GPU cluster operations: provisioning, capacity planning, and day-2 ownership.

NVIDIA ecosystem and CUDA — drivers, container runtime, toolkit compatibility, and their failure modes.

GPU scheduling, allocation, isolation, and utilisation optimisation across competing workloads.

Kubernetes GPU workloads: GPU Operator, device plugins, GPU-aware scheduling.

GPU troubleshooting and tuning: memory/OOM, driver faults, interconnect and throughput bottlenecks.

LLM inference and model serving in production (vLLM, Triton, KServe, Ray or equivalent) with demonstrated cost and performance gains.

Managed AI platform experience — SageMaker (training jobs, endpoints, inference components) or equivalent — alongside self-managed GPU serving.

Nice to Have:

SageMaker HyperPod, Bedrock, or Vertex AI at scale · DCGM and Prometheus/Grafana for GPU fleets · MIG and GPU partitioning in multi-tenant setups · NCCL and distributed GPU workloads · quantization and inference optimization (FP8/INT8, AWQ/GPTQ, speculative decoding, KV-cache tuning) · LLM observability and tracing (Langfuse, OpenTelemetry for LLMs) · eval frameworks and human-in-the-loop labeling · PyTorch profiling · GPU cluster schedulers and queueing · vector databases, RAG, LLM gateways, agentic systems · TypeScript/Express API development · React or other modern frontends · security or compliance-bound environments.

What Success Looks Like:

AI teams deploy and roll back models through self-service workflows, with no manual infrastructure work per deployment.

GPU utilisation is measured, forecast, and consistently optimised across the shared fleet.

Cost per token falls quarter over quarter, with the numbers on a dashboard.

Inference SLOs hold under peak enterprise load, and GPU incidents drop in frequency and time-to-resolve.

No model, prompt, or optimisation change reaches production without passing automated accuracy and quality evaluation.