Founding Head of Partnerships - Vals Smith

Vals, backed by Andreessen Horowitz, has launched to address AI model evaluation challenges. The startup aims to provide neutral benchmarking as enterprises struggle to compare competing AI claims from vendors using cherry-picked metrics. The company positions itself as an independent arbiter, focusing on real-world performance rather than synthetic test scores. This contrasts with existing benchmarking efforts often tied to specific vendors or academic institutions. The timing reflects growing enterprise frustration with AI performance claims. Companies reportedly waste millions deploying underperforming AI systems, whilst genuine innovations get overlooked. Vals faces competition from Nvidia's MLPerf benchmarks and evaluation methods from startups like Anthropic. The challenge will be achieving industry-wide adoption whilst maintaining neutrality amidst rapid model development and vendor interests.

Devin Fusion two-model harness cuts coding agent cost per task 11-46%: the counterintuitive case for pricier sidekicks. Cognition's Fusion harness pairs a frontier lead with a cheaper sidekick. Across 10 benchmark comparisons, costs fell 11-46% per task. A pricier sidekick can lower total session cost by reducing the lead model's review burden. Prices verified 2026-09-12. Cognition's Fusion agent harness pairs a frontier model lead with a cheaper sidekick for coding tasks. Across 10 published benchmark comparisons, costs fell 11-46% per task while scores moved by at most 6.4 points. The counterintuitive finding: a sidekick with higher token cost can reduce total session cost by reaching correct implementations in fewer turns, cutting the lead model's review burden. Prices verified 2026-09-12. Tl;dr. * Cognition partnered with Artificial Analysis and Vals AI to benchmark Fusion across 5 coding benchmarks and 2 model pairs (10 comparisons total). * On DeepSWE 1.1, Claude Fable 5.1 solo cost $14.63/task. With Fusion (Fable 5.1 + SWE-2), $7.88/task. That is a 46% cut with a 1.2-point score drop. * On Vals Code Migration, Fusion cost fell 41% and the score improved 2.7 points (54.6 to 57.3). * The GPT-6 Astra-led pairing was cheaper ($4.54/task average vs $7.90) but riskier: Terminal-Bench dropped 5.6 points, Vals Migration dropped 6.4. * Cognition observed that replacing Opus 4.8 with Fable 5 as lead (2x nominal price) lowered average session cost 9% because Fable delegated earlier and wrote clearer briefs. Verdict table. Agent harness Model pair Index Cost per task Tokens per task Time per task - - - - - - Devin Fusion CLI Fable 5.1 XHigh + SWE-2 Medium 62 $7.90 9.7M 35.8m Devin Fusion CLI Astra XHigh + SWE-2 Medium 59 $4.54 6.1M 24.7m Opencode [GLM-5.3](https://vynaris.com/models#glm-5-3) 54 $4.24 14.3M 48.1m Artificial Analysis Coding Agent Index v1.5 incorporates DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. Higher index is better. Cost is average API cost per task in USD. Savings across benchmarks. Fable 5.1 solo vs Fusion (Fable 5.1 + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 64.3 63.1 $14.63 $7.88 46% Terminal-Bench 4 57.6 56.1 $17.46 $13.37 23% SWE-Atlas QnA 64.8 65.9 $7.57 $5.00 34% Vals Code Migration 54.6 57.3 $70.97 $42.00 41% FrontierCode 1.1 63.6 63.5 $2.68 $1.67 38% Four of five Fable-led scores moved by 1.5 points or fewer. The migration score improved 2.7 points. This is the steadier pairing. GPT-6 Astra solo vs Fusion (Astra + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 67.6 67.3 $7.88 $4.69 40% Terminal-Bench 4 55.6 50.0 $10.08 $6.06 40% SWE-Atlas QnA 61.8 59.4 $5.72 $3.59 37% Vals Code Migration 67.7 61.3 $44.36 $35.51 20% FrontierCode 1.1 63.1 63.4 $2.62 $2.34 11% The Astra-led pairing is cheaper on average ($4.54 vs $7.90 per task) but produces larger quality swings. Terminal-Bench fell 5.6 points. Vals Code Migration fell 6.4 points. Two benchmarks improved slightly. The counterintuitive case for pricier sidekicks. The standard model routing approach picks a cheap or expensive model based on the initial prompt. Cognition argues this fails because the initial prompt does not reveal task difficulty. "Fix xyz bug" could be a one-line edge case or a full rearchitecture. Fusion takes a different approach: two agents run in parallel, each with its own persistent context window and prompt caching. The lead owns planning and review. The sidekick handles implementation. They exchange briefs and results, not full conversations. This keeps both caches warm. The counterintuitive finding concerns the sidekick. Cognition tested two sidekicks on the same Astra-led FrontierCode workload: Sidekick List price Score Cost per task - - - - [GPT-5.6 Luna](https://vynaris.com/models#gpt-5-6-luna) (high) $0.20/Mtok 62.0 $2.39 SWE-2 (medium) $0.75/Mtok (+275%) 63.4 $2.34 SWE-2 costs 275% more per token. The total session cost was 2% lower and the score was 1.4 points higher. A stronger sidekick reaches correct implementations in fewer turns, reducing the lead model's review and correction rounds. Cognition observed the same effect at the lead position. Replacing Opus 4.8 with Fable 5 as lead (2x nominal token cost) lowered average session cost 9% and scored higher on FrontierCode. Fable delegated earlier and wrote clearer briefs. Opus spent more turns directing the sidekick and redoing its work. This experiment used Fable 5, not Fable 5.1. Cognition's recommendation names Fable 5.1. Do not conflate the two. Price per completed task captures these interactions. Token cost alone does not. Put your own workload into the cost calculator to see how your model pair affects total session cost. The Astra quality swings. The Astra-led pairing is cheaper but riskier. The agent loop on Terminal-Bench dropped from 55.6 to 50.0 (a 5.6-point decline). Vals Code Migration dropped from 67.7 to 61.3 (6.4 points). These are not rounding errors. By contrast, the Fable-led pairing was steady. Four of five scores moved by 1.5 points or fewer. The one exception, Vals Code Migration, improved by 2.7 points. If you need consistent quality across task types, Fable-led is the safer choice. If you can absorb quality variance on some benchmarks, Astra-led delivers lower cost per task. Its coding agent token cost analysis showed 30x variance in token usage across coding agents. Fusion's cache hit rate of 96% on the Fable pairing helps explain why: keeping both caches warm avoids the re-reading costs that dominate long agent loops. The cache reads analysis covers the quadratic mechanics. When this harness does not help. Fusion adds a second model to every session. If your tasks are simple enough that a single cheaper model handles them in one pass, Fusion's overhead is pure cost. The FrontierCode benchmarks show the smallest savings (11-38%) because those tasks are shorter and involve less back-and-forth. The harness also assumes your workload benefits from delegation. If your tasks are single-turn (classification, extraction, simple completions), there is nothing to delegate. A frontier model solo or a cheap model solo will both be cheaper than running two agents. Finally, the published results are benchmark comparisons, not production data. Cognition and Artificial Analysis ran controlled evaluations on public benchmarks. Your repository, tool chain, and acceptance criteria will produce different numbers. Test on real work before adopting. The Fable 5.1 vs Opus 5 token efficiency analysis covers when Fable's higher token price pays off in solo mode. The LLM router estimation break-even covers when routing decisions are worth their inspection cost. Faq. Does Fusion work with any model pair? No. Cognition tunes the harness for each pair. Instructions that help one pair can hurt another. A new lead or sidekick requires its own delegation policy and benchmark pass. Is SWE-2 a standalone model I can call? SWE-2 is Cognition's sidekick model available within the Fusion harness. The benchmarks do not report standalone SWE-2 performance. What is the cheapest Fusion configuration? Astra + SWE-2 at $4.54/task average. But Opencode with GLM-5.3 is cheaper still at $4.24/task with a lower index (54 vs 59). The output tokens tell the story: Opencode uses 14.3M tokens per task versus Astra Fusion's 6.1M. Opencode is cheaper because GLM-5.3 has lower token prices, not because it is more efficient. Were these numbers independently verified? Artificial Analysis and Vals AI partnered with Cognition on the evaluation. The Cognition blog published the raw numbers. AlphaSignal independently covered the release. The cost figures are Cognition's, not third-party audited. Verification. All cost-per-task figures were fetched live from the Cognition blog and the Artificial Analysis comparison page on 2026-09-12. The AA Coding Agent Index v1.5 uses DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. The 10 comparisons span 5 benchmarks and 2 model pairs (Fable-led and Astra-led). Arithmetic was recomputed independently using Python's Decimal module. All percentage reductions match the source data to within 0.1 percentage points.

Vals AI built a test to see if AI models can create their own successors, and the results are fascinating. The Recursive Self-Improvement Index offers the first standardized way to measure whether AI systems can autonomously advance their own development. 2 hours ago Sponsored: CryptoSlots - Cryptoslots Play now! If you've ever wondered how close Crypto Briefing is to AI systems that can design better versions of themselves, Vals AI just built the scoreboard. The company's Recursive Self-Improvement Index, or RSI Index, attempts to quantify something that until now has lived mostly in the realm of theoretical worry: how capable are today's frontier models at conducting the research and development needed to build their successors? The top-performing model on the index, Claude Fable 5.1, scores 35.03%. Which sounds low until you understand the scale. How the RSI Index actually works. Vals AI designed the index around five specific tasks that mirror the real workflow of AI development. Models are evaluated under fixed compute and time constraints, meaning they can't just brute-force their way to a good score by burning through unlimited resources. The scoring system is anchored to reference points rather than arbitrary grades. A score of 0 represents a baseline, 0.5 maps to performance levels already documented in published research, and a theoretical optimum sits at 1. Claude Fable 5.1's score of 35.03% means it's performing meaningfully but still falls short of replicating techniques already known to researchers. The models can outperform existing techniques in experiment execution but generally lag in producing novel techniques. Why this matters now. The RSI Index launched in August 2026 through a collaboration between Vals AI and CoreWeave, timed alongside the company's $40 million Series A funding round. That round valued Vals AI at $400 million. CEO Rayan Krishnan has positioned the index as filling a critical gap. Major AI labs have already started referencing recursive self-improvement potential in their model cards, but without a standardized framework for comparison, those references lack a common basis for evaluation. In the six months following its funding round, Vals AI's revenue grew 8x compared to all of 2025. Its customer base doubled. Its staff tripled. The governance question. Krishnan has been vocal about what the RSI Index implies for regulation. The concern isn't that a model scoring 35% is about to go rogue. It's that the trajectory from 35% to higher scores could be steep, and the development of these capabilities is happening inside a handful of labs with limited external visibility. Krishnan has emphasized the need for international governance structures to keep pace with these advancements. The RSI Index gives it empirical grounding: it's harder to dismiss calls for oversight when you can point to a concrete metric showing measurable progress toward autonomous self-improvement. Disclosure: This article was edited by Editorial Team. For more information on how Crypto Briefing create and review content, see its Editorial Policy.