C

Crusoe

Builds AI data centers near cheap energy

Staff Engineer - Crusoe Managed Data Transfer, Distributed Systems

Software EngineeringConfirmed live in the last 24 hours
Full-timeSan Francisco, CA, USAIn-person

About the Job

Requirements

  • Hands-on experience designing and operating data-intensive distributed systems at scale, including production failure modes.
  • At least 8 years of experience shipping production software, including leading a service from prototype to general availability.
  • Understanding of object storage semantics, file systems, and TCP throughput, with the ability to profile and improve transfer performance.
  • Ability to evolve a live system toward a better design while keeping risk to current customers low.
  • Experience defining API boundaries and schema contracts with partner teams and negotiating them to agreement.
  • Experience carrying customer-facing on-call responsibility and building maintainable, observable, predictable systems.
  • Proficiency in Go or another modern compiled language; Go is strongly preferred.
  • Ability to explain architectural decisions to engineers, product managers, and engineering managers.

Responsibilities

  • Own the end-to-end design of Managed Data Transfer, including the job API and scheduler, Kubernetes transfer workers, storage integration, and network paths.
  • Evaluate the production prototype, decide what to refactor, extend, or replace, and migrate current users to the managed service without interruption.
  • Set the data-plane strategy, including sharding, concurrency, straggler handling, checksum verification, and benchmarks to demonstrate performance, aiming for line-rate transfers over dedicated interconnects.
  • Own interfaces between Managed Data Transfer and Storage, Network, Kubernetes, IAM, Console, Billing, and Monitoring; define partner-team contracts and ensure they are followed.
  • Design for additional cloud providers and Crusoe-internal transfers using a pluggable transfer engine and regional worker pools, and evaluate open-source and vendor options before building.
  • Set reliability, observability, and on-call standards for systems the team ships.
  • Work with Product on scope, pricing trade-offs, and milestones, and surface edge cases early enough to influence the plan.
  • Raise the team's technical level through design reviews, RFC feedback, and direct coaching.

Desired Qualifications

  • Experience building or operating bulk data transfer systems such as AWS DataSync, Google Storage Transfer Service, or rclone at petabyte scale.
  • Familiarity with cloud networking and interconnects such as AWS Direct Connect or GCP Interconnect.
  • Experience with cross-account cloud IAM and threat modeling for services that hold customer credentials.
  • In-depth knowledge of approximate nearest-neighbor indexing, including HNSW, IVF, product and scalar quantization, and DiskANN-style indexes stored on SSD.
  • Experience with AI, HPC, or Physical AI data pipelines.

About the Company

Crusoe

Builds AI data centers near cheap energy

Crusoe builds and runs data centers for artificial intelligence, placing them near cheap or otherwise wasted energy and handling everything from the power supply to the cloud software. It developed the flagship campus in Abilene, Texas, for Stargate, the AI infrastructure project backed by OpenAI and Oracle. AI companies rent its GPU servers on demand or under longer contracts through Crusoe Cloud to train and run their models, while larger customers lease entire campuses that Crusoe develops and operates for them. The company was founded in 2018, when it began running computers at oil fields on natural gas that would otherwise have been burned off, and it is based in Denver.

Company size
1,001-5,000
Company stage
Debt Financing
Total funding
$14.5B
Headquarters
Denver, CO
Founded
2018

Recently Posted Jobs

Explore more opportunities at Crusoe

See more jobs