B

Baseten

Runs AI models for production software

Software Engineer - Inference Product

Backend EngineeringPosted on Nov 19
Full-timeHybrid

About the Job

Requirements

  • A bachelor's, master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
  • At least 3 years of experience building and operating distributed systems, backend infrastructure, or large-scale APIs where reliability, latency, and scale are first-class concerns.
  • A proven track record of owning low-latency, reliable backend services, including rate limiting, authentication, quotas, metering, and migrations.
  • Infrastructure instincts with an understanding of performance, including profiling, tracing, capacity planning, and service-level objective management.
  • Ability to debug performance and reliability issues across multiple layers of the stack, from application behavior through runtime and infrastructure internals.
  • A strong sense of developer experience, considering how systems are used as well as how they work.
  • Eagerness to learn new languages, frameworks, and systems, and a genuine interest in inference engineering; prior machine learning or large language model experience is not required.
  • Written communication and collaboration skills, including writing clear design documents and working across functions.

Responsibilities

  • Build and operate the API layer behind Model APIs, including request handling, authentication, rate limiting, and routing, with high availability and low added latency.
  • Own usage metering and billing infrastructure, providing accurate, real-time accounting for every request and flexibility to support new pricing and packaging models.
  • Build rate-limiting and capacity systems that balance customer throughput against fleet utilization at scale and in real time.
  • Ship new products at the API layer that change how developers use models, including routing across models and features that make building on Model APIs easier.
  • Turn platform capabilities into a strong product experience through usage visibility, self-serve controls, and clear, predictable API behavior.
  • Instrument the edge with deep observability and alerting so issues are caught and fixed before customers notice.
  • Partner with the Model APIs, inference, and product teams to deliver new capabilities to customers quickly and safely.
  • Own projects end to end, from design through deployment, on-call, and iteration based on customer feedback.

Desired Qualifications

  • Deep Kubernetes experience, including operators and custom resources, plus familiarity with service meshes or API gateways.
  • Experience with distributed scheduling, autoscaling, or service orchestration.
  • A background in developer-facing infrastructure or APIs, or contributions to open-source infrastructure or machine learning systems.
  • Familiarity with observability tooling, continuous integration and continuous delivery systems, or release automation.
  • Experience with model serving or inference systems.

About the Company

Baseten

Runs AI models for production software

Baseten runs AI models for software companies that need fast, reliable responses inside their products, from text generation to speech and image processing. Its Inference Stack combines optimized model runtimes with infrastructure that deploys, serves, and monitors custom or open models in production. Teams package models with its open source Truss tool, while Model APIs let developers test ready-to-use models and Training runs jobs that adapt models to their own data. Customers pay for computing time or tokens processed, with negotiated plans for larger deployments and support needs. Baseten was founded in 2019 and is headquartered in San Francisco.

Company size
201-500
Company stage
Late Stage VC
Total funding
$4.1B
Headquarters
San Francisco, CA
Founded
2019

Recently Posted Jobs

Explore more opportunities at Baseten

See more jobs