FDA funds study on LLMs evaluating AI radiology reports. Cognita Imaging has won a $1.29 million agreement with the FDA to examine whether large language models can accurately assess AI-generated radiology reports. The 18-month study, which began June 22, targets a major hurdle in AI oversight: evaluating systems that produce open-ended text rather than simple diagnostic indicators. Current FDA-approved radiology AI tools handle specific tasks - such as identifying blood clots in CT scans or lung collapses in X-rays - making their assessment relatively simple. However, newer platforms, including those developed by Cognita, generate complete radiology reports containing hundreds of potential findings. This complexity renders traditional radiologist reviews impractical for large-scale use. The project will employ a panel of multiple LLMs to evaluate AI-generated reports against a dataset of 1 million patient exams collected from various U.S. healthcare settings. Performance will be measured across patient demographics, clinical environments, and imaging equipment types, including rare conditions. When the LLMs disagree on significant clinical issues, human radiologists will intervene to determine whether errors originate from the original AI, the LLM panel, or the baseline report. Akshay Chaudhari, Cognita's co-founder and a Stanford radiology associate professor, stated the initiative will quantify two primary risks in generative AI: hallucinations, plausible but incorrect information, and omissions of critical details. The aim is to assess whether these errors hold clinical significance and how frequently they appear. The final deliverables will include a full report for the FDA, along with open-source software code, guidelines for constructing LLM review panels, and comparisons between large and smaller validation datasets. Chaudhari stressed that the findings should provide a framework for future AI regulations, ensuring the approach remains adaptable beyond Cognita's own products. The FDA's increasing focus on AI regulation is evident in other recent actions. Earlier this year, Cognita received a breakthrough device designation for a vision-language model designed to interpret chest X-rays and produce draft radiology reports for physician review. This designation, granted separately from the research contract, reflects the agency's commitment to accelerating approvals for tools addressing critical clinical needs. Chaudhari noted that while the study plan for this model remains under development, the FDA's partnerships have clarified regulatory pathways. A separate FDA discussion paper on generative AI highlights broader efforts to establish standardized evaluation methods. The shift toward LLM-based review systems could reduce dependence on radiologist-led panels, which are slow and struggle to keep up with AI advancements. Yet challenges remain: if LLMs introduce biases or inaccuracies, the system's reliability depends on human oversight at key decision points. Founded in 2024 and acquired by Radiology Partners, the largest U.S. radiology group, just one year later, Cognita has drawn on insights from radiologists in both academic and community practices. Real-world pressures, such as radiologist shortages and mounting backlog cases, highlight why AI tools capable of automating report generation could address pressing workforce gaps. Chaudhari framed the project's success as dependent on proving that LLM juries can consistently match or improve upon traditional review methods without creating new risks. Seraphina Wentworth

Cognita Imaging receives $1.29 million FDA contract to test new approach to evaluating generative AI in radiology. The 18-month project will test an 'LLMs-as-a-jury' framework across approximately 1 million patient exams, with radiologists reviewing clinical discrepancies. Cognita Imaging has received a $1.29 million research contract from the US Food and Drug Administration (FDA) to test a new method for evaluating artificial intelligence (AI)-generated radiology reports. The contract took effect June 22, 2026, and will run for 18 months. Cognita, a wholly owned subsidiary of Mosaic Clinical Technologies, will use the funding for a project titled "Virtual Subject Matter Expert Agents for Radiology Report Evaluation." The research addresses difficulties associated with traditional reader studies, which require significant specialist time and often cannot cover the volume or diversity of imaging cases encountered in day-to-day clinical practice, according to the release. Under the agreement, Cognita plans to develop and validate a framework utilizing multiple large language models (LLMs) to assess both human-drafted and AI-generated radiology reports. The initiative is led by Akshay Chaudhari, PhD, Cognita co-founder, associate professor of radiology and biomedical data science at Stanford University, and principal investigator on the contract, along with Louis Blankemeier, PhD, co-founder and CEO of Cognita, who serves as co-investigator. "Once an AI system starts writing the entire report, the evaluation problem changes," says Chaudhari in a release. "A few hundred cases may tell you whether a model works in a narrow setting. They do not show every way it can fail in practice. We want to test whether a group of language models, with radiologists reviewing the difficult cases, can give us a clearer and more repeatable picture at real-world scale." Testing an 'llms-as-a-jury' framework at scale. The proposed methodology, termed "LLMs-as-a-jury," compares evaluations from several independent language models rather than relying on a single automated evaluator, according to the company. This multi-model approach is intended to enhance consistency, minimize single-model bias, and pinpoint areas where automated assessment can supplement expert review. Once developed, the framework will be deployed across approximately 1 million patient exams drawn from a diverse US cohort, according to Cognita. The analysis will assess model performance across various patient populations, clinical settings, imaging systems, and disease presentations, including rare pathologies that smaller datasets typically cannot capture. In addition, the research team will recreate smaller validation cohorts from the dataset to identify clinical information that might be overlooked in reduced-sample studies. When clinically significant disagreements occur, radiologists will examine the discrepancies to identify whether errors originated in the AI-generated report, the automated LLM jury, or the baseline radiologist report, according to the release. Cognita will apply these findings to refine the framework and explore how automated analysis and physician expertise can coordinate during premarket evaluation and post-market surveillance. "Radiologists know the edge cases matter," says Nina Kottler, MD, chief medical AI officer at Mosaic Clinical Technologies, in a release. "A model can look strong in a carefully selected study and still struggle at another hospital, on different equipment or with a rare presentation. This work lets us look for those differences at a scale that would be very difficult to reach through reader studies alone, while keeping radiologists involved where their judgment matters most." Regulatory deliverables and scope. The initiative expands on Cognita's previous development of radiology-focused language and vision-language systems, including GREEN, an open-source tool designed to detect clinically meaningful variations between reference reports and AI-generated text, the company stated. The project will also incorporate clinical workflows, infrastructure, and domain expertise from the Mosaic and Radiology Partners Clinical Services networks. Throughout the contract term, Cognita will supply the FDA with software code, implementation guidance for establishing LLM juries, comparative analyses of large and small validation cohorts, discrepancy reports reviewed by radiologists, and evaluations of findings and methodological limitations, according to the press release. The contract was granted via the FDA Broad Agency Announcement program for advanced research and development in regulatory science. Cognita noted that the agreement funds regulatory science research and does not constitute FDA clearance, approval, or endorsement of any proprietary technology or product.