Trust but verify: its first Public Sector Summit. Reflections from its first Public Sector Summit at STATION DC on youth wellbeing, AI diplomacy with China, and independent evaluation. Glenn Parham - 10/07/2026 On September 23, 2026, Vals AI, Inc. held its first Vals Public Sector Summit at STATION DC. It was great to bring researchers, young people, policy experts, and congressional staffers together to talk through three questions: How do Vals AI, Inc. protect kids using AI? What should Vals AI, Inc. be negotiating with China? And what should government expect from independent AI evaluators? I joined Vals because I'd kept running into the same problem in government. Vals AI, Inc. were being asked to use AI for consequential work, but Vals AI, Inc. needed an independent way to know whether it was actually ready for the mission. These are the kinds of conversations I wanted Vals AI, Inc. to have. The timing was especially interesting. Trump and Xi were scheduled to discuss AI the following day, and Treasury Secretary Scott Bessent and Chinese Vice Premier He Lifeng had just discussed a proposed notification channel for AI incidents with national security implications. Vals AI, Inc. had plenty to get into. Youth Wellbeing in the Age of AI. Caroline Figueroa and Andrea Mock kicked things off with their research on AI and youth mental health. One finding makes the need for evaluation pretty clear: a chatbot can give a helpful first answer and still become unsafe as the conversation continues. Testing one response won't tell you how that relationship develops. Their work was featured in The Washington Post that same day. Really proud to see this research getting attention, especially while policymakers are actively working through what protections should look like. Caroline then moderated a panel with Marissa Edmund, Jake Barr, Sparkle Rainey, and Kanika Mehra. Kanika shared an example that made the conversation feel very concrete: her friendship with her downstairs neighbor. Driving her to the airport, lending coffee creamer, listening after a bad date. Those small obligations are part of what makes a relationship meaningful. An AI companion doesn't ask any of that from you. Marissa raised another practical question. If Vals AI, Inc. require chatbots to remind users they're talking to AI, will those warnings actually help? Or will they become another cookie banner everyone ignores? These are things Vals AI, Inc. can test. Do safeguards hold up over a longer conversation? Do they interrupt unhealthy patterns? Does the chatbot help a young person reach someone who can actually support them? Jake also had useful advice for researchers: get your work in front of congressional staff while they're still developing an idea, before it becomes a bill. Even sending over a relevant paper can start a conversation. As Sparkle put it, "We should be proactive and not wait till it's too late." Negotiating AI with China. I moderated this panel with Kevin Wolf, Ryan Fedasiuk, and Karson Elmgren. It was great to have people who could get into the mechanics of these issues, from export controls to how a communication channel would actually work. Kevin started with the objective. What national security outcome are Vals AI, Inc. trying to achieve with a restriction? Ryan made an important distinction between leading on benchmarks and winning global adoption. Freely available Chinese models create a different competitive challenge. When I asked how Vals AI, Inc. could verify an agreement, Ryan pushed Vals AI, Inc. to take a step back: "What do we want from China?" It's a basic question, but Vals AI, Inc. need a clear answer before deciding what to measure or enforce. Karson had one of my favorite practical observations of the day. An AI crisis hotline might work better as an email inbox, or even a fax. The Chinese officials staffing it may need approval before they can respond. A channel that requires an immediate conversation might not fit how their system works. I think that's worth keeping in mind as these talks continue. Vals AI, Inc. need specific commitments, evidence that they're being met, and a way for the people involved to actually communicate. There's a lot of work between announcing a dialogue and having something useful during an incident. Government's Role in Evaluating AI. Independent evaluators have been getting a lot of attention on Capitol Hill, including through the bipartisan FRONTIER Act. It was especially great to have Will Burns from Rep. Jay Obernolte's office and Dylan Irlbeck from Rep. Lori Trahan's office, who helped shape the bill, join Rayan Krishnan for its closing panel. The proposal gets specific about what evaluators would do: ongoing assessments of the largest developers, access to unredacted records, and notifying Commerce within 72 hours of identifying imminent catastrophic risk. For Vals AI, Inc. at Vals, that raises questions Vals AI, Inc. need to take seriously. Can Vals AI, Inc. get the evidence Vals AI, Inc. need? Can Vals AI, Inc. reach its own conclusions and communicate findings a company might not want to hear? And how should someone check that Vals AI, Inc. is doing its job well? Rayan put it plainly in his opening remarks: "Our credibility has to come from the quality of the work and our willingness to be clear about the model's limits." That standard applies to Vals AI, Inc., too. Bringing the research into the room. Vals AI, Inc. also had posters displaying its cybersecurity, child-safety, and environmental-impact research. Its environmental work was recently covered by Bloomberg, and it was great to bring these projects into the room with people shaping AI policy. Thanks again to its speakers, the STATION DC team, everyone at Vals who helped make this happen, and everyone who joined Vals AI, Inc.. Would love to keep these conversations going, especially about what you need evaluated and how Vals AI, Inc. can make the research useful for your work.

Vals, backed by Andreessen Horowitz, has launched to address AI model evaluation challenges. The startup aims to provide neutral benchmarking as enterprises struggle to compare competing AI claims from vendors using cherry-picked metrics. The company positions itself as an independent arbiter, focusing on real-world performance rather than synthetic test scores. This contrasts with existing benchmarking efforts often tied to specific vendors or academic institutions. The timing reflects growing enterprise frustration with AI performance claims. Companies reportedly waste millions deploying underperforming AI systems, whilst genuine innovations get overlooked. Vals faces competition from Nvidia's MLPerf benchmarks and evaluation methods from startups like Anthropic. The challenge will be achieving industry-wide adoption whilst maintaining neutrality amidst rapid model development and vendor interests.

Devin Fusion two-model harness cuts coding agent cost per task 11-46%: the counterintuitive case for pricier sidekicks. Cognition's Fusion harness pairs a frontier lead with a cheaper sidekick. Across 10 benchmark comparisons, costs fell 11-46% per task. A pricier sidekick can lower total session cost by reducing the lead model's review burden. Prices verified 2026-09-12. Cognition's Fusion agent harness pairs a frontier model lead with a cheaper sidekick for coding tasks. Across 10 published benchmark comparisons, costs fell 11-46% per task while scores moved by at most 6.4 points. The counterintuitive finding: a sidekick with higher token cost can reduce total session cost by reaching correct implementations in fewer turns, cutting the lead model's review burden. Prices verified 2026-09-12. Tl;dr. * Cognition partnered with Artificial Analysis and Vals AI to benchmark Fusion across 5 coding benchmarks and 2 model pairs (10 comparisons total). * On DeepSWE 1.1, Claude Fable 5.1 solo cost $14.63/task. With Fusion (Fable 5.1 + SWE-2), $7.88/task. That is a 46% cut with a 1.2-point score drop. * On Vals Code Migration, Fusion cost fell 41% and the score improved 2.7 points (54.6 to 57.3). * The GPT-6 Astra-led pairing was cheaper ($4.54/task average vs $7.90) but riskier: Terminal-Bench dropped 5.6 points, Vals Migration dropped 6.4. * Cognition observed that replacing Opus 4.8 with Fable 5 as lead (2x nominal price) lowered average session cost 9% because Fable delegated earlier and wrote clearer briefs. Verdict table. Agent harness Model pair Index Cost per task Tokens per task Time per task - - - - - - Devin Fusion CLI Fable 5.1 XHigh + SWE-2 Medium 62 $7.90 9.7M 35.8m Devin Fusion CLI Astra XHigh + SWE-2 Medium 59 $4.54 6.1M 24.7m Opencode [GLM-5.3](https://vynaris.com/models#glm-5-3) 54 $4.24 14.3M 48.1m Artificial Analysis Coding Agent Index v1.5 incorporates DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. Higher index is better. Cost is average API cost per task in USD. Savings across benchmarks. Fable 5.1 solo vs Fusion (Fable 5.1 + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 64.3 63.1 $14.63 $7.88 46% Terminal-Bench 4 57.6 56.1 $17.46 $13.37 23% SWE-Atlas QnA 64.8 65.9 $7.57 $5.00 34% Vals Code Migration 54.6 57.3 $70.97 $42.00 41% FrontierCode 1.1 63.6 63.5 $2.68 $1.67 38% Four of five Fable-led scores moved by 1.5 points or fewer. The migration score improved 2.7 points. This is the steadier pairing. GPT-6 Astra solo vs Fusion (Astra + SWE-2). Benchmark Solo score Fusion score Solo cost Fusion cost Cost cut - - - - - - DeepSWE 1.1 67.6 67.3 $7.88 $4.69 40% Terminal-Bench 4 55.6 50.0 $10.08 $6.06 40% SWE-Atlas QnA 61.8 59.4 $5.72 $3.59 37% Vals Code Migration 67.7 61.3 $44.36 $35.51 20% FrontierCode 1.1 63.1 63.4 $2.62 $2.34 11% The Astra-led pairing is cheaper on average ($4.54 vs $7.90 per task) but produces larger quality swings. Terminal-Bench fell 5.6 points. Vals Code Migration fell 6.4 points. Two benchmarks improved slightly. The counterintuitive case for pricier sidekicks. The standard model routing approach picks a cheap or expensive model based on the initial prompt. Cognition argues this fails because the initial prompt does not reveal task difficulty. "Fix xyz bug" could be a one-line edge case or a full rearchitecture. Fusion takes a different approach: two agents run in parallel, each with its own persistent context window and prompt caching. The lead owns planning and review. The sidekick handles implementation. They exchange briefs and results, not full conversations. This keeps both caches warm. The counterintuitive finding concerns the sidekick. Cognition tested two sidekicks on the same Astra-led FrontierCode workload: Sidekick List price Score Cost per task - - - - [GPT-5.6 Luna](https://vynaris.com/models#gpt-5-6-luna) (high) $0.20/Mtok 62.0 $2.39 SWE-2 (medium) $0.75/Mtok (+275%) 63.4 $2.34 SWE-2 costs 275% more per token. The total session cost was 2% lower and the score was 1.4 points higher. A stronger sidekick reaches correct implementations in fewer turns, reducing the lead model's review and correction rounds. Cognition observed the same effect at the lead position. Replacing Opus 4.8 with Fable 5 as lead (2x nominal token cost) lowered average session cost 9% and scored higher on FrontierCode. Fable delegated earlier and wrote clearer briefs. Opus spent more turns directing the sidekick and redoing its work. This experiment used Fable 5, not Fable 5.1. Cognition's recommendation names Fable 5.1. Do not conflate the two. Price per completed task captures these interactions. Token cost alone does not. Put your own workload into the cost calculator to see how your model pair affects total session cost. The Astra quality swings. The Astra-led pairing is cheaper but riskier. The agent loop on Terminal-Bench dropped from 55.6 to 50.0 (a 5.6-point decline). Vals Code Migration dropped from 67.7 to 61.3 (6.4 points). These are not rounding errors. By contrast, the Fable-led pairing was steady. Four of five scores moved by 1.5 points or fewer. The one exception, Vals Code Migration, improved by 2.7 points. If you need consistent quality across task types, Fable-led is the safer choice. If you can absorb quality variance on some benchmarks, Astra-led delivers lower cost per task. Its coding agent token cost analysis showed 30x variance in token usage across coding agents. Fusion's cache hit rate of 96% on the Fable pairing helps explain why: keeping both caches warm avoids the re-reading costs that dominate long agent loops. The cache reads analysis covers the quadratic mechanics. When this harness does not help. Fusion adds a second model to every session. If your tasks are simple enough that a single cheaper model handles them in one pass, Fusion's overhead is pure cost. The FrontierCode benchmarks show the smallest savings (11-38%) because those tasks are shorter and involve less back-and-forth. The harness also assumes your workload benefits from delegation. If your tasks are single-turn (classification, extraction, simple completions), there is nothing to delegate. A frontier model solo or a cheap model solo will both be cheaper than running two agents. Finally, the published results are benchmark comparisons, not production data. Cognition and Artificial Analysis ran controlled evaluations on public benchmarks. Your repository, tool chain, and acceptance criteria will produce different numbers. Test on real work before adopting. The Fable 5.1 vs Opus 5 token efficiency analysis covers when Fable's higher token price pays off in solo mode. The LLM router estimation break-even covers when routing decisions are worth their inspection cost. Faq. Does Fusion work with any model pair? No. Cognition tunes the harness for each pair. Instructions that help one pair can hurt another. A new lead or sidekick requires its own delegation policy and benchmark pass. Is SWE-2 a standalone model I can call? SWE-2 is Cognition's sidekick model available within the Fusion harness. The benchmarks do not report standalone SWE-2 performance. What is the cheapest Fusion configuration? Astra + SWE-2 at $4.54/task average. But Opencode with GLM-5.3 is cheaper still at $4.24/task with a lower index (54 vs 59). The output tokens tell the story: Opencode uses 14.3M tokens per task versus Astra Fusion's 6.1M. Opencode is cheaper because GLM-5.3 has lower token prices, not because it is more efficient. Were these numbers independently verified? Artificial Analysis and Vals AI partnered with Cognition on the evaluation. The Cognition blog published the raw numbers. AlphaSignal independently covered the release. The cost figures are Cognition's, not third-party audited. Verification. All cost-per-task figures were fetched live from the Cognition blog and the Artificial Analysis comparison page on 2026-09-12. The AA Coding Agent Index v1.5 uses DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA. The 10 comparisons span 5 benchmarks and 2 model pairs (Fable-led and Astra-led). Arithmetic was recomputed independently using Python's Decimal module. All percentage reductions match the source data to within 0.1 percentage points.