C

CoreWeave

Rents GPU cloud computing for AI developers

Senior Software Engineer 2 - Inference

Applied Machine LearningPosted on Nov 13
Full-timeIn-person

About the Job

Requirements

  • IC3 candidates need approximately 3–5 years of industry experience; IC4 candidates need approximately 5–8 years of industry experience building distributed systems or cloud services.
  • Strong coding ability in Python or Go; C++ is a plus.
  • Deep familiarity with networked systems and performance.
  • Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks including Prometheus, Grafana, and OpenTelemetry.
  • Practical knowledge of inference internals, including batching, caching, mixed precision (BF16/FP8), and streaming token delivery.
  • A proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work.
  • A bachelor's or master's degree in Computer Science or a related field, or equivalent practical experience.

Responsibilities

  • Lead design reviews and drive architecture within the team; decompose multi-service work into clear milestones.
  • Define and own SLIs/SLOs; ensure post-incident actions are completed and reliability improves release over release.
  • Implement advanced optimizations, such as micro-batch schedulers, speculative decoding, and KV-cache reuse, and quantify their impact.
  • Strengthen incident readiness through capacity planning, autoscaling policy, graceful degradation, and rollback/traffic-shift strategies.
  • Mentor IC1/IC2 engineers; review cross-team designs and elevate coding and testing standards.
  • For IC4, own an area spanning multiple services and teams, such as request routing and adaptive scheduling, cost-per-token analytics, or GPU resource isolation.

Desired Qualifications

  • Contributions to inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, or TorchServe.
  • Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
  • Experience leading multi-team initiatives or partnering with customers on mission-critical launches.

About the Company

CoreWeave

Rents GPU cloud computing for AI developers

CoreWeave rents cloud computing to AI labs and businesses that train models and run them in production for research and commercial applications. Its platform combines NVIDIA graphics chips, storage, networking, and software built for demanding AI workloads across connected clusters scaled to fit each project. CoreWeave Forge connects tools for tracking experiments, testing models, and improving AI systems using feedback from production. Customers rent computing capacity through committed contracts or usage charges, while managed services handle the infrastructure needed to keep large workloads running. CoreWeave was founded in 2017 and is headquartered in Livingston, New Jersey.

Company size
1,001-5,000
Company stage
IPO
Headquarters
Livingston, NJ
Founded
2017

Recently Posted Jobs

Explore more opportunities at CoreWeave

See more jobs