Inactive
V

Vals AI

Evaluates AI models via professional benchmarks

Evaluations Engineer

Natural Language Processing & LLMsPosted on Jul 21
Full-timeSan Francisco, CA, USA

About the Job

Requirements

  • Familiarity with the current leading large language models, their relative performance, and practical use of large language models.
  • Strong engineering fundamentals and a track record of building and shipping significant-scope projects at work, in side projects, or in open source.
  • Significant professional experience with Python.
  • Experience working in development sprints, Git workflows, and pull request reviews.
  • Willingness to work long hours during model releases and deliver high-quality results under tight deadlines.
  • Ability to learn new skills and technologies quickly.
  • Initiative and autonomy to build where needed.
  • A solution-oriented approach to hard problems.

Responsibilities

  • Evaluate new large language model releases across the Vals AI suite of benchmarks.
  • Work directly with open-source and closed-source foundation model labs to evaluate model performance.
  • Use Docent to analyze common failure modes and patterns in model performance.
  • Work with the social media team to post findings and results.
  • Add new models and maintain integrations in the model library.
  • Improve and maintain the infrastructure used to run agentic and non-agentic benchmarks.

Desired Qualifications

  • Previous experience benchmarking large language models or creating benchmarks.
  • Previous experience working at a startup or starting a company.
  • Technical writing experience and ability.
  • Machine learning research experience.

About the Company

Vals AI

Evaluates AI models via professional benchmarks

Vals AI provides benchmarks and evaluation infrastructure to measure AI models on professional tasks across law, finance, engineering, and healthcare. Its platform uses privately held test sets and automated scoring so enterprises can compare models and assess their AI products. The Vals Smith product creates custom coding benchmarks by aggregating tasks from GitHub repositories to test programming ability. By offering private test sets and domain-specific evaluation, it helps enterprises choose models and validate AI-powered products with transparent, reproducible results.

Company size
11-50
Company stage
Series A
Total funding
$40M
Headquarters
San Francisco, CA
Founded
2023

Recently Posted Jobs

Explore more opportunities at Vals AI

See more jobs