R

Ramp

Runs corporate cards and finance workflows

Research Engineer - Benchmarks & Evaluation

AI ResearchConfirmed live in the last 24 hours
Full-time$204.4k - $352k /yrHybrid

About the Job

Requirements

  • Hands-on experience building large language model benchmarks or evaluations used to make real decisions.
  • Strong Python and software engineering skills, including writing reproducible and fast evaluation code.
  • Statistical judgment, including variance, confidence intervals, sample size, inter-rater agreement, and distinguishing meaningful changes from noise.
  • Working knowledge of post-training, including supervised fine-tuning and reinforcement learning with verifiable rewards, and how evaluations feed into post-training.
  • Comfort with SQL and large datasets, including working directly from production traces.

Responsibilities

  • Build benchmarks from real workflows by turning agent traces, customer edge cases, and expert labels into versioned benchmark suites for long-horizon agents running for hours across tools and systems, as well as tool use, multi-step reasoning, and financial judgment.
  • Own the benchmarking effort by deciding what to measure, running benchmark suites across models and agent versions, and reporting results the team uses to make decisions.
  • Design graders using rubrics, programmatic checks, and large language model-as-judge approaches calibrated against human agreement, with a clear understanding of where each approach breaks.
  • Close the loop with post-training by turning evaluations into reward functions and reinforcement learning environments, and catching reward hacking and contamination before they ship.
  • Publish external benchmarks and write-ups on how models perform at real finance work.

Desired Qualifications

  • Industry experience on an evaluation, benchmarking, or post-training team.
  • Experience building reinforcement learning environments or verifiers for agentic tasks.
  • Published benchmark or evaluation research, such as NeurIPS Datasets & Benchmarks, ICML, or ACL.
  • Experience building or contributing to evaluation frameworks such as verifiers, lm-evaluation-harness, or HELM.
  • Familiarity with accounting, finance, or other expert-domain work.
  • Public work that can be reviewed, such as a benchmark, evaluation suite, or failure-analysis write-up.

About the Company

Ramp

Runs corporate cards and finance workflows

Ramp runs a financial operations platform for businesses and their finance teams, linking corporate cards with expense records and invoice payments. Finance teams use its software to collect receipts, approve spending and send transaction data to their accounting systems, while employees submit expenses through connected apps. Its Router service also directs AI requests across outside model providers, selecting models based on cost and performance. Customers use a free core plan or pay for broader software automation features, with partner banks issuing cards and providing deposit services. Ramp was founded in 2019 and has its headquarters in New York City.

Company size
1,001-5,000
Company stage
Growth Equity (Venture Capital)
Total funding
$5.4B
Headquarters
New York City, New York
Founded
2019

Recently Posted Jobs

Explore more opportunities at Ramp

See more jobs