Principal Engineer - Performance and Benchmarking
$206k - $333k+ Discretionary bonus + Equity awards
Expert
Bellevue, WA, USA+1 moreMore locations: Sunnyvale, CA, USA
In Person
QA & TestingPerformance Testing
About the job
Requirements
- At least 10 years of experience building distributed systems or high-performance computing/cloud services, with deep expertise in large-scale machine learning training or similar high-performance workloads.
- A proven track record of architecting or building planet-scale data systems, such as telemetry platforms, observability stacks, cloud data warehouses, or large-scale online analytical processing engines.
- Deep understanding of graphics processing unit performance, including CUDA, NCCL, RDMA, NVLink/PCIe, and memory bandwidth; model-server stacks such as Triton, vLLM, TensorRT-LLM, and TorchServe; and distributed training frameworks such as PyTorch FSDP, DeepSpeed, and Megatron-LM.
- Proficiency with Kubernetes and machine learning control planes, with familiarity with SUNK, Kueue, and Kubeflow in production environments.
- Ability to communicate with executives, customers, auditors, and open-source communities.
Responsibilities
- Define the multi-year benchmarking strategy and roadmap; prioritize models and workloads, including large language models, diffusion, vision, and speech, as well as hardware tiers.
- Build, lead, and mentor a high-performing team of performance engineers and data analysts.
- Establish governance for benchmark claims, including documented methodologies, versioning, reproducibility, and audit trails.
- Lead end-to-end MLPerf Inference and Training submissions, including workload selection, cluster planning, runbooks, audits, and publication of results.
- Coordinate optimization tracks with NVIDIA using CUDA, cuDNN, TensorRT/TensorRT-LLM, Triton, and NCCL to achieve competitive results, and drive upstream fixes where needed.
- Design a Kubernetes-native, repeatable benchmarking service that exercises CoreWeave stacks across SUNK, Kueue, and Kubeflow pipelines.
- Measure and report p50/p95/p99 latency, jitter, tokens per second, time-to-first-token, cold-start/warm-start, and cost per token/request across models, precisions (BF16/FP8/FP4), batch sizes, and GPU types.
- Maintain a corpus of representative scenarios, including streaming, batch, and multi-tenant scenarios, and datasets; automate comparisons across software releases and hardware generations.
- Build continuous integration and continuous delivery pipelines and Kubernetes controllers/operators to schedule benchmarks at scale.
- Integrate benchmarking with observability stacks, including Prometheus, Grafana, and OpenTelemetry, and results warehouses.
- Implement supply-chain integrity for benchmark artifacts, including software bills of materials and Cosign signatures.
- Partner with NVIDIA, key independent software vendors, and open-source projects, including vLLM, Triton, KServe, PyTorch/DeepSpeed, and ONNX Runtime, to co-develop optimizations and upstream improvements.
- Support Sales and Sales Engineers with authoritative numbers for requests for proposals and competitive evaluations; brief analysts and press with rigorous, defensible data.
Desired Qualifications
- Experience with time-series databases, log-structured merge trees, or custom storage engine development.
- Experience running MLPerf submissions (Inference and/or Training) or equivalent audited benchmarks at scale.
- Contributions to MLPerf, Triton, vLLM, PyTorch, KServe, or similar open-source projects.
- Experience benchmarking multi-region fleets and large clusters of thousands of GPUs.
- Publications or talks on machine-learning performance, latency engineering, or large-scale benchmarking methodology.
About the company
CoreWeave rents cloud computing to AI labs and businesses that train models and run them in production for research and commercial applications. Its platform combines NVIDIA graphics chips, storage, networking, and software built for demanding AI workloads across connected clusters scaled to fit each project. CoreWeave Forge connects tools for tracking experiments, testing models, and improving AI systems using feedback from production. Customers rent computing capacity through committed contracts or usage charges, while managed services handle the infrastructure needed to keep large workloads running. CoreWeave was founded in 2017 and is headquartered in Livingston, New Jersey.
Company Size
1,001-5,000
Company Stage
IPO
Headquarters
Livingston, NJ
Founded
2017
Get referred to CoreWeave
See people who can refer or advise you
Simplify's Take
What believers are saying
- Demand is compounding: August 2026 results showed CoreWeave quarterly revenue more than doubled to $2.58 billion.
- Contracts support expansion: CoreWeave reported $104 billion of June 2026 backlog, subject to service delivery.
- Eligible U.S. employees buy discounted equity: CoreWeave's purchase plan prices shares 15% below the lower opening or closing price.
What critics are saying
- Customer losses threaten stability: three buyers supplied 72% of CoreWeave's Q2 2026 revenue.
- Borrowing consumes revenue: CoreWeave's net interest expense reached $640 million in Q2 2026, reported in August 2026.
- Afton technicians face urgent callouts: CoreWeave requires onsite emergency response within 60 minutes during scheduled on-call shifts.
What makes CoreWeave unique
- Hardware access stays direct: CoreWeave runs Kubernetes on bare metal, without a virtualization layer.
- Support skips nontechnical handoffs: CoreWeave says customers reach engineers cross-trained across its stack directly.
- Top AI labs use CoreWeave: the company says nine of ten leading foundation model providers rely on its cloud.
Help us improve and share your feedback! Did you find this helpful?
Benefits
- Health Insurance
- Dental Insurance
- Vision Insurance
- Life Insurance
- Disability Insurance
- Health Savings Account/Flexible Spending Account
- Tuition Reimbursement
- Mental Health Support
- Family Planning Benefits
- Paid Parental Leave
- Hybrid Work Options
- 401(k) Company Match