V

Vals AI

Independent AI model evaluation platform

Evaluations Engineer

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeSan Francisco, CA, USA

About the Job

Requirements

  • Familiarity with the current leading large language models, their relative performance, and practical use of large language models.
  • Strong engineering fundamentals and a track record of quickly building and shipping significant-scope projects in jobs, side projects, or open source.
  • Significant professional experience with Python.
  • Experience working in development sprints, Git workflows, and pull request reviews.
  • Willingness to work long hours during model releases and deliver high-quality results under tight deadlines.
  • Ability to work in person in San Francisco.

Responsibilities

  • Evaluate new large language model releases across the Vals AI suite of benchmarks.
  • Work directly with open-source and closed-source foundation model labs to evaluate model performance.
  • Use Docent to analyze common failure modes and patterns in model performance.
  • Work with the social media team to post findings and results.
  • Add new models and maintain integrations in the model library.
  • Help improve and maintain the infrastructure used to run agentic and non-agentic benchmarks.

Desired Qualifications

  • Previous experience benchmarking large language models or creating benchmarks.
  • Previous experience working at a startup or starting a company.
  • Technical writing experience and ability.
  • Machine learning research experience.

About the Company

Vals AI

Independent AI model evaluation platform

Vals AI provides independent evaluation and benchmarking for large language models and AI applications with domain-specific leaderboards. It tests models on real-world tasks using private datasets to prevent test leakage and reports metrics such as tool use, multi-modality, long-context reasoning, accuracy, latency, and cost, while offering an evaluation framework for private labs and enterprise teams and public benchmarks like Finance Agent and Valkyrie. The company differentiates itself by delivering unbiased, practitioner-focused measurements built with domain experts and by focusing on economically valuable, complex professional tasks rather than academic problems. Its goal is to become the trusted measurement layer for the AI economy, helping research labs and enterprises make informed decisions about AI model adoption and deployment.

Company size
11-50
Company stage
Series A
Total funding
$40M
Headquarters
San Francisco, CA
Founded
2023

Recently Posted Jobs

Explore more opportunities at Vals AI

See more jobs