V

Vals AI

Independent AI model evaluation platform

Head of Research

Full-Time
$225k - $275k/yr+ Equity + $1,500 housing stipend
Expert
PhD
San Francisco, CA, USA
In Person
AI & Machine LearningAI Research

About the job

Requirements
  • A PhD in machine learning or natural language processing, in progress or completed, or an equivalent industry research track record.
  • Deep familiarity with the large language model evaluation landscape, including existing benchmarks, their failure modes, judge-model approaches, and human-in-the-loop methodologies.
  • A bias toward research that affects what people actually deploy rather than benchmarks that are easy to game.
  • Strong written and verbal communication to publish, present, and talk to customers and labs.
  • Ability to work in person in San Francisco.
Responsibilities
  • Advance the science of evaluation by developing new paradigms beyond judge models, human-in-the-loop methods, and static benchmarks for long-horizon, real-world tasks.
  • Oversee the broader research portfolio and set direction across ongoing and future projects.
  • Publish research that moves the field forward and is cited.
  • Recruit and grow a research team alongside the founders.
  • Work directly with enterprise customers and lab partners on their evaluation problems.
Desired Qualifications
  • A widely cited benchmark or evaluation framework that the candidate has built or co-built.
  • Prior experience at a frontier lab such as Anthropic, OpenAI, Google DeepMind, or Meta FAIR, or at a research-led startup.
  • Domain depth in one or more verticals: legal, finance, insurance, or healthcare.
  • Experience leading or mentoring other researchers.
  • A public research presence through papers, blog posts, talks, or recognized open-source contributions.

About the company

V

Vals AI

View
Website

Vals AI provides independent evaluation and benchmarking for large language models and AI applications. It builds domain-specific leaderboards and benchmarks (finance, law, healthcare, software) using private datasets to prevent test leakage, and it measures performance on real-world tasks and capabilities like tool use, multi-modality, reasoning, accuracy, latency, and cost. The platform offers an evaluation infrastructure for private labs and enterprises to test their own data and tasks, and it also maintains public benchmarks such as Finance Agent and the Valkyrie evaluation framework. Its goal is to give neutral, practitioner-focused metrics that help research labs and enterprise teams choose and deploy AI models with confidence.

Company Size

11-50

Company Stage

Series A

Total Funding

$40M

Headquarters

San Francisco, California

Founded

2023

Get referred to Vals AI

See people who can refer or advise you

Simplify Jobs

Simplify's Take

What believers are saying

  • a16z led Vals’ $40 million Series A on August 13, 2026.
  • Vals launched Vals Smith, Vals 2.0, and CoreWeave-backed RSI Index in August 2026.
  • Revenue rose 8x in six months; customers doubled and staff tripled.

What critics are saying

  • OpenAI, Anthropic, and Google can replace Vals with internal evals by 2027.
  • Public trust collapses if vendors game hidden tests or benchmark contamination surfaces.
  • MLPerf, Artificial Analysis, and model cards commoditize scoring, squeezing Vals into a feature.

What makes Vals AI unique

  • Private held-out datasets block leakage and preserve benchmark signal across model families.
  • Domain experts shape finance, legal, healthcare, and coding tasks for real workloads.
  • Vals Index weights finance, coding, and legal by U.S. GDP, not academic trivia.

Help us improve and share your feedback! Did you find this helpful?

Benefits

  • Health Insurance
  • Dental Insurance
  • 401(k) Retirement Plan
  • Unlimited Paid Time Off
  • Meal Benefits
  • Company Equity
  • Relocation Assistance

Growth & Insights and Company News

Headcount

6 month growth

-3%

1 year growth

-15%

2 year growth

3%
Epium Ltd
Oct 15th, 2026
Vals AI plans independent tests of model routers.

Vals AI plans independent tests of model routers. Vals AI is developing an independent evaluation of model routers, which select which AI model handles each request based on quality, cost, and latency. The evaluation will measure live, end-to-end agent performance across finance, coding, legal, and tax work, with results due on a public leaderboard soon. Performance differences across models create an opportunity for routing. Vals reports that five different models lead its eight benchmarks, with no model placing in the top three across all of them. A theoretical oracle that selects the perfect model for each task in hindsight would outperform the best single model by 14 points while costing about 40% less. That calculation illustrates potential gains, not demonstrated router performance. Long-running workflows make those gains harder to capture. A cheaper model can introduce errors that require additional calls, while switching models can sacrifice prompt-cache savings. Evaluations based on isolated public questions can miss these effects and face risks from saturated benchmarks or prior exposure to test data. Vals plans to consult each router on every model call while keeping prompts, tools, and graders fixed. Products will run with their default model pools and settings, with configurations documented. Task outcomes, cost, cache usage, and completion time will be compared with the best and cheapest individual models in each router's pool, using the oracle as a reference. The Vals Index includes mostly private, expert-written benchmarks to reduce contamination risk.

TechBuzz AI
Sep 19th, 2026
Vals secures A16z funding to build neutral AI benchmarking platform

Vals, backed by Andreessen Horowitz, has launched to address AI model evaluation challenges. The startup aims to provide neutral benchmarking as enterprises struggle to compare competing AI claims from vendors using cherry-picked metrics. The company positions itself as an independent arbiter, focusing on real-world performance rather than synthetic test scores. This contrasts with existing benchmarking efforts often tied to specific vendors or academic institutions. The timing reflects growing enterprise frustration with AI performance claims. Companies reportedly waste millions deploying underperforming AI systems, whilst genuine innovations get overlooked. Vals faces competition from Nvidia's MLPerf benchmarks and evaluation methods from startups like Anthropic. The challenge will be achieving industry-wide adoption whilst maintaining neutrality amidst rapid model development and vendor interests.

Vynaris
Sep 12th, 2026
Devin Fusion two-model harness cuts coding agent cost per task 11-46%: the counterintuitive case for pricier sidekicks.

Devin Fusion two-model harness cuts coding agent cost per task 11-46%: the counterintuitive case for pricier sidekicks. Cognition's Fusion harness pairs a frontier lead with a cheaper sidekick. Across 10 benchmark comparisons, costs fell 11-46% per task. A pricier sidekick can lower total session cost by reducing the lead model's review burden. Prices verified 2026-09-12. Cognition's Fusion agent harness pairs a frontier model lead with a cheaper sidekick for coding tasks. Across 10 published benchmark comparisons, costs fell 11-46% per task while scores moved by at most 6.4 points. The counterintuitive finding: a sidekick with higher token cost can reduce total session cost by reaching correct implementations in fewer turns, cutting the lead model's review burden. Prices verified 2026-09-12. Tl;dr. * Cognition partnered with Artificial Analysis and Vals AI to benchmark Fusion across 5 coding benchmarks and 2 model pairs (10 comparisons total). * On DeepSWE 1.1, Claude Fable 5.1 solo cost $14.63/task. With Fusion (Fable 5.1 + SWE-2), $7.88/task. That is a 46% cut with a 1.2-point score drop. * On Vals Code Migration, Fusion cost fell 41% and the score improved 2.7 points (54.6 to 57.3). * The GPT-6 Astra-led pairing was cheaper ($4.54/task average vs $7.90) but riskier: Terminal-Bench dropped 5.6 points, Vals Migration dropped 6.4. * Cognition observed that replacing Opus 4.8 with Fable 5 as lead (2x nominal price) lowered average session cost 9% because Fable delegated earlier and wrote clearer briefs. Verdict table. Agent harness Model pair Index Cost per task Tokens per task Time per task - - - - - - Devin Fusion CLI Fable 5.1 XHigh + SWE-2 Medium 62 $7.90 9.7M 35.8m Devin Fusion CLI Astra XHigh + SWE-2 Medium 59 $4.54 6.1M 24.7m Opencode [GLM-5.3](https://vynaris.com/models#glm-5-3) 54 $4.24 14.3M 48.1m Artificial Analysis Coding Agent Index v1.5 incorporates DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. Higher index is better. Cost is average API cost per task in USD. Savings across benchmarks. Fable 5.1 solo vs Fusion (Fable 5.1 + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 64.3 63.1 $14.63 $7.88 46% Terminal-Bench 4 57.6 56.1 $17.46 $13.37 23% SWE-Atlas QnA 64.8 65.9 $7.57 $5.00 34% Vals Code Migration 54.6 57.3 $70.97 $42.00 41% FrontierCode 1.1 63.6 63.5 $2.68 $1.67 38% Four of five Fable-led scores moved by 1.5 points or fewer. The migration score improved 2.7 points. This is the steadier pairing. GPT-6 Astra solo vs Fusion (Astra + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 67.6 67.3 $7.88 $4.69 40% Terminal-Bench 4 55.6 50.0 $10.08 $6.06 40% SWE-Atlas QnA 61.8 59.4 $5.72 $3.59 37% Vals Code Migration 67.7 61.3 $44.36 $35.51 20% FrontierCode 1.1 63.1 63.4 $2.62 $2.34 11% The Astra-led pairing is cheaper on average ($4.54 vs $7.90 per task) but produces larger quality swings. Terminal-Bench fell 5.6 points. Vals Code Migration fell 6.4 points. Two benchmarks improved slightly. The counterintuitive case for pricier sidekicks. The standard model routing approach picks a cheap or expensive model based on the initial prompt. Cognition argues this fails because the initial prompt does not reveal task difficulty. "Fix xyz bug" could be a one-line edge case or a full rearchitecture. Fusion takes a different approach: two agents run in parallel, each with its own persistent context window and prompt caching. The lead owns planning and review. The sidekick handles implementation. They exchange briefs and results, not full conversations. This keeps both caches warm. The counterintuitive finding concerns the sidekick. Cognition tested two sidekicks on the same Astra-led FrontierCode workload: Sidekick List price Score Cost per task - - - - [GPT-5.6 Luna](https://vynaris.com/models#gpt-5-6-luna) (high) $0.20/Mtok 62.0 $2.39 SWE-2 (medium) $0.75/Mtok (+275%) 63.4 $2.34 SWE-2 costs 275% more per token. The total session cost was 2% lower and the score was 1.4 points higher. A stronger sidekick reaches correct implementations in fewer turns, reducing the lead model's review and correction rounds. Cognition observed the same effect at the lead position. Replacing Opus 4.8 with Fable 5 as lead (2x nominal token cost) lowered average session cost 9% and scored higher on FrontierCode. Fable delegated earlier and wrote clearer briefs. Opus spent more turns directing the sidekick and redoing its work. This experiment used Fable 5, not Fable 5.1. Cognition's recommendation names Fable 5.1. Do not conflate the two. Price per completed task captures these interactions. Token cost alone does not. Put your own workload into the cost calculator to see how your model pair affects total session cost. The Astra quality swings. The Astra-led pairing is cheaper but riskier. The agent loop on Terminal-Bench dropped from 55.6 to 50.0 (a 5.6-point decline). Vals Code Migration dropped from 67.7 to 61.3 (6.4 points). These are not rounding errors. By contrast, the Fable-led pairing was steady. Four of five scores moved by 1.5 points or fewer. The one exception, Vals Code Migration, improved by 2.7 points. If you need consistent quality across task types, Fable-led is the safer choice. If you can absorb quality variance on some benchmarks, Astra-led delivers lower cost per task. Its coding agent token cost analysis showed 30x variance in token usage across coding agents. Fusion's cache hit rate of 96% on the Fable pairing helps explain why: keeping both caches warm avoids the re-reading costs that dominate long agent loops. The cache reads analysis covers the quadratic mechanics. When this harness does not help. Fusion adds a second model to every session. If your tasks are simple enough that a single cheaper model handles them in one pass, Fusion's overhead is pure cost. The FrontierCode benchmarks show the smallest savings (11-38%) because those tasks are shorter and involve less back-and-forth. The harness also assumes your workload benefits from delegation. If your tasks are single-turn (classification, extraction, simple completions), there is nothing to delegate. A frontier model solo or a cheap model solo will both be cheaper than running two agents. Finally, the published results are benchmark comparisons, not production data. Cognition and Artificial Analysis ran controlled evaluations on public benchmarks. Your repository, tool chain, and acceptance criteria will produce different numbers. Test on real work before adopting. The Fable 5.1 vs Opus 5 token efficiency analysis covers when Fable's higher token price pays off in solo mode. The LLM router estimation break-even covers when routing decisions are worth their inspection cost. Faq. Does Fusion work with any model pair? No. Cognition tunes the harness for each pair. Instructions that help one pair can hurt another. A new lead or sidekick requires its own delegation policy and benchmark pass. Is SWE-2 a standalone model I can call? SWE-2 is Cognition's sidekick model available within the Fusion harness. The benchmarks do not report standalone SWE-2 performance. What is the cheapest Fusion configuration? Astra + SWE-2 at $4.54/task average. But Opencode with GLM-5.3 is cheaper still at $4.24/task with a lower index (54 vs 59). The output tokens tell the story: Opencode uses 14.3M tokens per task versus Astra Fusion's 6.1M. Opencode is cheaper because GLM-5.3 has lower token prices, not because it is more efficient. Were these numbers independently verified? Artificial Analysis and Vals AI partnered with Cognition on the evaluation. The Cognition blog published the raw numbers. AlphaSignal independently covered the release. The cost figures are Cognition's, not third-party audited. Verification. All cost-per-task figures were fetched live from the Cognition blog and the Artificial Analysis comparison page on 2026-09-12. The AA Coding Agent Index v1.5 uses DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. The 10 comparisons span 5 benchmarks and 2 model pairs (Fable-led and Astra-led). Arithmetic was recomputed independently using Python's Decimal module. All percentage reductions match the source data to within 0.1 percentage points.

Crypto Briefing
Sep 9th, 2026
Vals AI built a test to see if AI models can create their own successors, and the results are fascinating.

Vals AI built a test to see if AI models can create their own successors, and the results are fascinating. The Recursive Self-Improvement Index offers the first standardized way to measure whether AI systems can autonomously advance their own development. 2 hours ago Sponsored: CryptoSlots - Cryptoslots Play now! If you've ever wondered how close Crypto Briefing is to AI systems that can design better versions of themselves, Vals AI just built the scoreboard. The company's Recursive Self-Improvement Index, or RSI Index, attempts to quantify something that until now has lived mostly in the realm of theoretical worry: how capable are today's frontier models at conducting the research and development needed to build their successors? The top-performing model on the index, Claude Fable 5.1, scores 35.03%. Which sounds low until you understand the scale. How the RSI Index actually works. Vals AI designed the index around five specific tasks that mirror the real workflow of AI development. Models are evaluated under fixed compute and time constraints, meaning they can't just brute-force their way to a good score by burning through unlimited resources. The scoring system is anchored to reference points rather than arbitrary grades. A score of 0 represents a baseline, 0.5 maps to performance levels already documented in published research, and a theoretical optimum sits at 1. Claude Fable 5.1's score of 35.03% means it's performing meaningfully but still falls short of replicating techniques already known to researchers. The models can outperform existing techniques in experiment execution but generally lag in producing novel techniques. Why this matters now. The RSI Index launched in August 2026 through a collaboration between Vals AI and CoreWeave, timed alongside the company's $40 million Series A funding round. That round valued Vals AI at $400 million. CEO Rayan Krishnan has positioned the index as filling a critical gap. Major AI labs have already started referencing recursive self-improvement potential in their model cards, but without a standardized framework for comparison, those references lack a common basis for evaluation. In the six months following its funding round, Vals AI's revenue grew 8x compared to all of 2025. Its customer base doubled. Its staff tripled. The governance question. Krishnan has been vocal about what the RSI Index implies for regulation. The concern isn't that a model scoring 35% is about to go rogue. It's that the trajectory from 35% to higher scores could be steep, and the development of these capabilities is happening inside a handful of labs with limited external visibility. Krishnan has emphasized the need for international governance structures to keep pace with these advancements. The RSI Index gives it empirical grounding: it's harder to dismiss calls for oversight when you can point to a concrete metric showing measurable progress toward autonomous self-improvement. Disclosure: This article was edited by Editorial Team. For more information on how Crypto Briefing create and review content, see its Editorial Policy.

Venture5
Aug 16th, 2026
Vals AI raises $40M Series A

Vals AI. Vals AI raises $40M Series A. Vals AI, a San Francisco-based AI agent testing platform, raised $40 million in a Series A fundraising round led by Andreessen Horowitz a16z with participation from 8VC and Pear VC.