Production Engineer - LLM Serving

Production Engineer, LLM Serving

Location: Seoul, South Korea (Hybrid)

About the Job

Owns the reliability, observability, and service-level performance of production LLM serving services on FuriosaAI's RNGD NPUs. You will improve latency, throughput, and resource efficiency through workload-driven measurement, bottleneck analysis, and software engineering.

Our stack uses llm-d and Furiosa-LLM, with Istio for traffic management in Kubernetes-based deployments and Mooncake for L3 KV-cache storage. You will own the serving service layer, collaborating with infrastructure, inference engine, runtime, and compiler teams on improvements that span their components.

Key Responsibilities

• Operates and improves RNGD-based LLM serving services, defining SLIs/SLOs and production readiness criteria to guide reliability and capacity decisions.
• Builds metrics, logs, traces, dashboards, and actionable alerts across the serving request path, tracking availability, errors, time to first token (TTFT), inter-token and end-to-end latency, throughput, and NPU utilization.
• Designs benchmarks and load tests reflecting production input/output lengths, concurrency, and traffic patterns. Uses latency distributions, throughput, and utilization to establish baselines, detect regressions, and plan serving capacity.
• Diagnoses bottlenecks across routing, queueing, batching, caching, networking, and inference execution. Improves service components and configurations to increase throughput and resource efficiency while meeting latency SLOs.
• Improves deployment topologies, autoscaling, health checks, graceful degradation, and Istio traffic policies for serving workloads, partnering with the infrastructure team on shared Kubernetes and service mesh capabilities.
• Automates serving deployments and validates changes through performance tests, canary rollouts, and rollback procedures, collaborating with the build and release team on release artifacts.
• Participates in on-call and incident response, from mitigation to root-cause analysis and preventive improvements. Builds runbooks and recovery automation to reduce operational toil.

Minimum Qualifications

• Bachelor's degree in Computer Science or a related field, or equivalent practical experience, with 3+ years developing and operating production services.
• Strong programming skills in Rust, C++, Python, or Go, with the ability to read, instrument, and improve existing production code.
• Hands-on experience operating services in Kubernetes and solid understanding of Linux, containers, networking, concurrency, and distributed systems.
• Understanding of LLM serving concepts, including prefill/decode, batching, KV-cache management, and request scheduling, and their impact on performance.
• Experience profiling production services and analyzing telemetry to identify bottlenecks across the request path and validate improvements in tail latency, throughput, and resource efficiency.
• Experience building observability systems and using metrics, logs, and traces to diagnose production incidents, restore services, and implement preventive improvements.
• Ability to communicate quantitative findings and collaborate across teams to deliver measurable production improvements.

Preferred Qualifications

• Experience operating LLM serving systems using llm-d, vLLM, SGLang, TensorRT-LLM, or similar technologies.
• Experience optimizing request routing, load balancing, queueing, or autoscaling, and translating workload measurements into capacity plans or serving-cost estimates.
• Experience with inference optimizations such as continuous batching, prefix KV-cache reuse, speculative decoding, distributed inference, or kernel optimization.
• Experience with Istio traffic management, infrastructure as code, or GitOps, including automated rollout and rollback workflows.
• Experience with Prometheus, Grafana, OpenTelemetry, and reliability practices such as SLO design, error budgets, and post-incident reviews.
• Experience operating GPU/NPU workloads in on-premises or cloud environments and diagnosing issues across drivers, runtime, orchestration, and applications.
• Experience developing software in Rust.