T

Together AI

Runs cloud infrastructure for open AI models

Staff / Senior Staff AI Inference Reliability Engineer - Compute Infra

Site Reliability EngineeringConfirmed live in the last 24 hours
Full-timeSan Francisco, CA, USAIn-person

About the Job

Requirements

  • Experience operating a provisioning or host-lifecycle pipeline in production, troubleshooting stuck or failed state transitions across discovery, bring-up, GPU driver/CUDA validation, health checks, and decommission/RMA at fleet scale, and translating failure patterns into state-machine design improvements.
  • Deep experience owning production reliability for large-scale distributed systems.
  • Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in multi-node training or disaggregated-inference contexts, including reading InfiniBand port counters and subnet manager logs.
  • Familiarity with operational realities of multi-tenant model serving, including LoRA/adapter-based serving and differentiated SLOs across model pools of different priority.
  • A track record of eliminating operational toil through self-built automation and tooling.
  • Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
  • Comfort with event-driven architectures, backpressure/circuit-breaking design, and building observability through metrics, logs, and traces.
  • A software-engineering mindset, with the ability to write and ship production code and engineer GPU infrastructure as software.

Responsibilities

  • Operate the provisioning state machine in production and own the health of the host-lifecycle pipeline, including discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommission/RMA; triage stuck or failed fleet-scale transitions and feed failure patterns into state and transition design.
  • Triage and extend automated self-healing by diagnosing new failure modes—including InfiniBand issues in disaggregated prefill/decode serving, GPU Xid variants, and hero-model or LoRA-specific faults—driving fixes to production, and implementing detection/remediation plugins.
  • Own end-to-end reliability for provisioning and fleet health, including SLOs and error budgets; lead incident response when idempotency, retry, rollback, or drift assumptions fail, and drive systemic fixes.
  • Partner with the inference/ML platform team from the production side, sharing how cluster topologies, interconnects, and scheduling constraints fail and turning those failure modes into platform requirements.
  • Work with machine learning researchers, engineers, and product managers to bring technologies into production and support research through engineering.
  • Introduce techniques, tools, and architecture to improve model-inference performance, latency, throughput, and efficiency.
  • Build tools to expose bottlenecks and instability, then design and implement solutions to priority issues.
  • Lead reliability work across infrastructure, ML platform, and product teams; build alignment on shared priorities, manage stakeholders through incidents and roadmap trade-offs, and represent reliability in planning.

Desired Qualifications

  • GPU/ML-serving infrastructure experience is a strong plus.
  • InfiniBand/RDMA troubleshooting experience in a multi-node training or disaggregated-inference context is preferred.

About the Company

Together AI

Runs cloud infrastructure for open AI models

Together AI runs cloud infrastructure for developers and businesses that train, customize, and deploy AI models, with a focus on open models that customers control. Its platform combines model hosting with tools for adapting models to tasks such as coding assistants, customer support, and image generation. Developers call models through an API, reserve dedicated capacity for steady workloads, or rent GPU clusters for training, while the company charges for model usage and computing resources. Teams also fine-tune models on their own data and deploy the results through the same platform without building a separate serving system. Together AI was founded in 2022 and is based in San Francisco.

Company size
201-500
Company stage
Series C
Total funding
$1.3B
Headquarters
San Francisco, CA
Founded
2022

Recently Posted Jobs

Explore more opportunities at Together AI

See more jobs