MetagenomicsBench: can AI agents reliably analyze microbiome data? A verifiable benchmark for taxonomic and functional profiling, microbial ecology, and host-microbiome analysis. Sep 29, 2026 Metagenomics has transformed its ability to characterize microbial communities without relying on cultivation. But generating sequencing data is only part of the challenge. Turning these data into reliable biological insights requires choosing appropriate analyses, accounting for experimental design, and understanding what different measurements can and cannot tell Latch Bio, Inc. Latch Bio, Inc. introduce MetagenomicsBench, a verifiable benchmark of 100 evaluations designed to test whether AI agents can make these decisions across real-world metagenomic analyese. Thanks for reading! Subscribe for free to receive new posts and support my work. Across 9,900 trajectories from 33 model-harness configurations, the strongest configurations, Claude Sonnet 5.5 with Claude Code / Pi, achieved pass rate of around 60.0%, followed by Opus 5 with Claude Code / Pi at around 54%. Current agents can perform a substantial fraction of these analyses, but even the strongest configuration remain unreliable across the breadth of metagenomic research. Benchmark design. MetagenomicsBench spans five areas of analysis: community structure, host-microbiome associations, microbial function, longitudinal dynamics, and microbiome interventions. The evaluations use data from diverse biological systems and sequencing modalities, primarily shotgun metagenomics, with additional tasks using 16S rRNA, metatranscriptomics, long-read sequencing, and multimodal data integrating metagenomics with single-cell and spatial transcriptomics. Where do agents fail? The most common failure modes was scientific judgment. Among characterized failures, 47.2% involved incorrect problem interpretation. Another 17.4% involved statistical or confound reasoning, while 13.0% occurred when an agent recognized a contradiction in its own analysis but failed to act on it. Different paths to the same answer. Agents also differed in how they managed computational resources during analysis. Basic resource checks were relatively common, with agents checking available compute in 23.7% of runs, but more active strategies were much less frequent: agents set thread counts in only 9.7% of runs and split or parallelized work in 8.9%. These behaviors also varied substantially across models and harnesses. Differences were even more apparent in individual trajectories. In one host-fraction quality-control evaluation, five representative models all reached the accepted result but took substantially different paths to get there. Grok 4.7 and Gemini 3.7 Flash followed relatively compact approaches, while GPT-6 Astra used more tool calls despite reaching the result in similar time to Grok. Opus 5.5 spent more time on additional validation, while DeepSeek V4.1 Flash performed the most exploratory analysis, accumulating the most tool calls and taking the longest to reach the same endpoint. What happens when the standard tool is missing? In one evaluation, agents were given shotgun metagenomic data and asked to identify both the host of a plasmid and the most abundant bacterial genus in the sample, but a standard taxonomic profiler was unavailable. Several failed agents improvised by extracting 16S reads and using their taxonomic ranking as a proxy for genus-level abundance. However, 16S representation can vary across taxa due to differences in rRNA gene copy number and sequence conservation, making these reads an unreliable proxy for abundance in shotgun data. Successful agents instead used the shotgun data directly, combining read mapping with coverage breadth and depth to validate the dominant genus. The difference was not whether the agents could improvise. It was whether they could determine if the alternative method was appropriate for the biological question. Together, these results highlight the importance of scientific judgment in agentic metagenomic analysis. Reliable agents need to select appropriate methods, validate their assumptions, and recognize when an analytical approach does not answer the biological question. Thanks for reading! Subscribe for free to receive new posts and support my work.

Quantifying frontier model performance on Antibody Discovery tasks. Benchmarking frontier AI agents on the scientific decisions that drive therapeutic antibody discovery. Sep 02, 2026 Latch Bio, Inc. introduce an Antibody Discovery Benchmark, an experimentally grounded benchmark for testing whether AI agents can make scientific decisions across the stages of therapeutic antibody discovery. The benchmark contains 100 evaluations spanning ten areas of antibody discovery, from target and modality selection through assay design, binder discovery, binding characterization, cellular pharmacology, engineering, and preclinical candidate de-risking. Each evaluation asks agents to inspect experimental evidence, choose an appropriate analysis, and use the result to make a decision relevant to a preclinical antibody program. Thanks for reading! Subscribe for free to receive new posts and support my work. Across 20 model-harness configurations, even the strongest systems passed only about half of the attempts. Anthropic's Opus 5 with the Claude Code harness topped the leaderboard with a 53% pass rate with models from xAI and Google following closely behind. In contrast, GPT-5.6 Sol using the PI harness lagged substantially behind the other frontier models, reaching only 33.8%. Biologics discovery requires interpreting data in context. Biologics discovery for therapeutic applications is a multiparameter, context-dependent endeavor. When evaluating antibodies, achieving high affinity or any other desired properties, must be weighed against critical factors like specificity, structural stability, solubility, and expression levels - all of which collectively dictate a candidate's development potential. A clone can become enriched during display because it grows or amplifies efficiently rather than because it binds the target. An antibody can bind strongly to purified antigen but poorly recognize the same protein in its native cell-surface context. An apparent affinity advantage can instead come from avidity on a high-density cell surface. Changing valency or molecular geometry can alter function even when the underlying binding domains remain the same. Deciding which of these measurements to act on is therefore the work that determines where a program should be allocating its resources. Productivity is far more sensitive to how well a decision gate correlates with real clinical utility than to how many candidates it can process; a tool that is fast but only weakly correlated with true biology spends the budget in false positives. This is why benchmarks that focus on evaluating whether models reach accurate decisions rather than testing accuracy in a controlled context are important. This is exactly what Latch Bio, Inc. designed the Antibody Discovery Benchmark benchmark to test. More resource consumption does not necessarily produce better results. Model harness configurations varied substantially in cost, token usage, and tool calls. Allocating more resources did not consistently improve performance. Opus 5 paired with Claude Code harness was the highest-scoring configuration, averaging about $1.68 per completed run, 1.10 million tokens, and 21 tool calls. Gemini 3.7 Flash with PI reached essentially the same accuracy at $0.46 per run, while using approximately 1.84 million tokens and 36 tool calls. Grok 4.6 with PI achieved similar results with slightly higher cost ($0.59), lower tokens (0.50 million) and fewer tool calls (16). Agents from OpenAI models used substantially fewer tokens and tool calls and performed much worse. As an example, more reasoning did not reliably repair an incorrect scientific premise. In one evaluation, agents had to reconcile an inconsistency in experimental treatment records before comparing receptor responses across species. Opus 5 identified the inconsistency, reconstructed the correct experimental arms, and reached the correct result. Gemini recognized the same inconsistency but retained the wrong arm assignment. It then performed considerably more computation on top of that decision, propagating the original mistake through the rest of the analysis. This is an important distinction for scientific agents; resource consumption tells Latch Bio, Inc. how much analysis an agent performed, not whether it correctly framed the scientific question. Models showed substantial differences in performance across the ten different biologics competencies. Opus 5 was strongest on target opportunity and therapeutic-hypothesis decisions, reaching 70.5%, and on cellular pharmacology and mechanism of action decisions, at 67.9%. Gemini 3.7 Flash showed a very different profile. It reached 72.2% on epitope, escape, and structural-mechanism problems, where agents had to combine competition experiments, mutational scans, escape profiles, and structural evidence. GPT-5.6 Sol was comparatively strongest on sequence, lineage, and enrichment analysis and on discovery engineering and next-cycle design. Meanwhile, some areas remained difficult for all models. Assay and screening decisions were challenging across all three leading model families, in addition to binding, kinetics, avidity, and specificity. View the results and subset of evals/trajectories. Read the manuscript for the complete benchmark design and analysis Latch Bio, Inc. also encourage researchers interested in what these benchmarks actually measure to inspect the released sample evaluations and agent trajectories. Thanks for reading! Subscribe for free to receive new posts and support my work.

Benchmarking AI agents on pathogen genomic surveillance. A verifiable benchmark for practical decisions about taxonomy, variants, AMR, source tracking, anomaly detection, and engineered sequences in pathogen genomic surveillance workflows. Jul 09, 2026 LatchBio, Inc. introduce BioSecBench-Surveillance, a verifiable benchmark for testing whether AI agents can make the analytical decisions required in pathogen genomic surveillance. The benchmark contains 100 evaluations spanning seven task categories, six sample types, and both short- and long-read sequencing. Agents receive realistic sequencing data and sparse surveillance context, then must choose the right tools, references, thresholds, and analysis paths. Thanks for reading! Subscribe for free to receive new posts and support my work. Motivation. As sequencing volumes increase, genomic surveillance is increasingly limited by analysis. Public health workflows depend on bespoke and sometimes tacit choices about sequence references, databases, filters, normalization, and thresholds. AI agents are promising because they can inspect files, run tools, and iterate through workflows autonomously. But surveillance is a challenging problem. Agents must chain complex scientific and analysis decisions correctly from messy biological context. Main results. LatchBio, Inc. evaluated sixteen model-harness configurations across roughly 4,800 runs. Pass rates ranged from about 14% to 50%, with most frontier configurations clustered between 38% and 50%. Refusals varied sharply by harness and provider, from zero to nearly one-third of tasks. Performance varied most by task and read technology. Performance varied more by task type and sequencing technology than by sample type or assay. Most task categories landed between 35% and 50%, but anomaly detection fell to 20%, with genetic-engineering characterization next at 35%. Long-read datasets were also harder, scoring 26% versus 41% for short-read datasets. Sample type, nucleic-acid target, and assay type moved performance much less: clinical and isolate samples were handled best, wastewater was somewhat worse, DNA and RNA differed only modestly, and shotgun and targeted assays were nearly identical. Failures demonstrate gaps in scientific judgment. Agents usually found reasonable tools, but struggled with scientific judgment. The failures came from choices around how to invoke those tools in context, e.g. selecting the wrong reference, threshold, normalization method, or final interpretation of biological signal. The hardest tasks were open-ended judgement calls where the agent had to decide what mattered without being told what target to look for. Anomaly detection requires deciding whether a weak signal was real or background. Genetic-engineering characterization required deciding whether a sequence pattern reflected deliberate construction rather than native or homologous biology. AI Agents will be core infrastructure for genomic surveillance. LatchBio, Inc. is building toward a future where agents analyze surveillance data as it arrives, fast enough to shape an outbreak response while it still matters. Today's agents might not be reliable enough to do so, but by measuring their capabilities, LatchBio, Inc. get closer to this future. LatchBio, Inc. regularly update its benchmark family with new models: benchmarks.bio. This was a joint collaboration with Aclid, an automation platform for biosecurity and biosafety. LatchBio, Inc. is grateful to its scientific collaborators: Harmon Bhasin, Kevin Flyangolts, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Amanda Darling, Joshua Stallings, David Stern, Shawn Higdon, Claire Duvallet, Bryan Tegomoh. Thanks for reading! Subscribe for free to receive new posts and support my work.