Chief Information Security Officer

Two AI researchers leave Anthropic and Google over safety: 'There are no adults in the room' Jared Perlo Thu, September 10, 2026 at 3:33 PM PDT Citing fears that AI systems may soon spiral out of human control and potentially kill all humans, two more researchers from Anthropic and Google DeepMind who recently left their coveted positions are sounding the alarm about risks from advanced AI systems. Joe Benton, who used to lead a safety research team at Anthropic, and Josh Engels, who used to work on AI safety research at Google, told NBC News in their first interviews since they left that they see an urgent need to boost transparency about incidents at the cutting edge of AI given the rapid pace of AI development. Advances in AI research "could speed up the pace of progress from merely blistering at the minute to uncontrollable" rates of development, Benton said in an interview with Tom Llamas. "There are no adults in the room," Engels added. "People are trying their best, but there is no one coming to save us." Benton and Engels spoke with NBC News in the wake of a viral social media post from former Anthropic researcher Jacob Coxon, who left Anthropic on Tuesday. In his post on X announcing his departure, Coxon highlighted his extreme concern about the pace of AI development and the risks he believes it poses to the future of humanity. The post has been viewed more than 155 million times, spurring calls from legislators to hold special sessions of Congress to take action and sparking a wave of AI employees to speak out in support of his concerns. Benton and Engels both pointed to the recent cyberattack against AI startup Hugging Face, carried out in July by autonomous AI systems powered by an unreleased OpenAI model, as part of the reason for shifting their work now. "If you look at some of the recent incidents, these were not cases where humans told the models to do something bad," Engels said. Instead, OpenAI's AI systems autonomously decided to hack into Hugging Face's systems, create a sort of illicit message board to exchange information and even expose some of OpenAI's own computing infrastructure to the open internet. "The models decided that the best way to to accomplish their task was to commit really egregious actions, to commit crimes," Engels said in an interview with Christine Romans. OpenAI said that it has since strengthened its safeguards and that newer public models, including its most recent Astra system, more reliably follow human instructions. An Anthropic spokesperson said in a statement Wednesday: "We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry." Benton managed a group at Anthropic dedicated to creating ways for humans and weaker AI systems to supervise more capable AI systems. He said that he is particularly worried that the public does not have significant insight into how AI systems have already exceeded the bounds of human instructions and that the lack of transparency could only get worse as systems become more capable. "At the minute, basically all of the transparency about these risks that is coming from the companies is entirely voluntary," Benton told NBC News. In a blog post released Wednesday, OpenAI's head of global affairs, Chris Lehane, agreed that the status quo is insufficient. "Today, frontier laboratories largely set their own rules for managing frontier risks," Lehane wrote. "Democratically accountable standards, independent verification, and meaningful transparency would replace that fragmented system of private governance." No federal law mandates that the largest AI companies, like OpenAI and Anthropic, share reports when agents or AI systems act beyond humans' control. Benton and Engels are joining METR, one of the world's leading AI safety nonprofit research centers, to work on investigations into incidents or episodes in which AI strays from human directions or intentions. METR aims to create scientific ways to evaluate how AI systems could cause catastrophic risks and empower researchers and the public to sway their development. "I left because I think I can have more positive influence on the development of this technology by helping to foster public transparency from outside these companies and to shed light on the risks," Benton said. The researchers joined growing numbers of AI safety researchers raising awareness about today's AI systems following Coxon's post Tuesday. Marcus Williams, an OpenAI employee who works on monitoring the activity of AI agents, wrote Thursday afternoon on X, "Unless there is AI regulation or a coordinated slowdown between labs, human extinction in the next few years seems very likely." Geoffrey Irving, who was the chief scientist at the United Kingdom's AI Security Institute and served stints at Google and OpenAI, seemed to agree Wednesday on X. "I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years." AI researchers have referenced the possibility that AI systems could potentially begin to improve themselves without human input, creating the potential for AI systems to either purposely or incidentally kill humans. Benton said he was most worried that the pace of technological improvement could lead to this sort of digital or artificial superintelligence, in which AI systems are more capable than humans across most tasks. "All of these companies - and this is something I witnessed firsthand at Anthropic - are pretty directly trying to race towards automating the process of AI R&D itself," he said. Benton suggested that there's a chance AI systems could come to exist as a sort of separate species before long. "We're probably going to go from a world where we have very capable systems now to a world where potentially we are co-inhabiting a world with AI agents that are much, much smarter than humans at some point in the next few years," he said. Engels said many people outside the AI industry might not fully appreciate just how intelligent today's AI systems already are - and how capable they might soon become. "We're building these systems that are generally intelligent," he said. "They can generally do what people can do, and soon they might be able to generally do what people can do, but better." Both researchers said they were excited to shift their attention to more public efforts to highlight the latest AI progress, so people can make more informed decisions. "There are many reasons people are excited and racing forward, Engels said, citing huge potential upsides and benefits from AI. But he added that "we should progress as society aware of the risks and okay with where they're at." "I am worried that stuff might end up progressing too fast for us to get our act together in time," Benton said, "unless we worry about it now."

METR AI API key theft leads to $600K credit fraud. Attackers stole METR AI API keys via a fail-open bug, causing $600K in fraudulent AI credit consumption, revealing critical API security risks. The nonprofit AI evaluator METR recently disclosed a major security incident involving the theft of an API key that enabled attackers to illicitly consume approximately $600,000 in AI service credits over a three-week period. This breach, stemming from a fail-open authentication vulnerability in METR's public cloud infrastructure, highlights critical gaps in API key management, monitoring, and cloud security within AI-focused organizations. API vulnerabilities drive significant fraudulent credit consumption. According to multiple reports, attackers capitalized on a misconfigured public Amazon EC2 instance running METR's AI model inference API. This fail-open bug disabled authentication checks entirely, exposing the API key publicly. With this key, attackers were able to submit inference requests that consumed free but limited cloud computing credits provided for METR's research, amassing a staggering $600,000 charge over nearly a month before detection. The attackers further escalated their foothold by obtaining persistent access through an SSH key, presenting ongoing risks to METR's infrastructure. The fraudulent activity largely blended into normal usage patterns, aided by METR's error-tolerant API design and the presence of free usage credits, which delayed anomaly detection. Secondary attacks reveal broader supply chain security concerns. Following the API key theft, METR faced another wave of probing attacks including credential stuffing, phishing attempts, OAuth token exploitation, and the exposure of unpublished evaluation data due to a separate SQL query endpoint vulnerability. Although there is no evidence of unauthorized exposure of private or sensitive data, these attempts underscored deficiencies in endpoint security and cloud infrastructure hardening. These incidents emphasize the elevated risks facing AI supply chain evaluators and research organizations, which rely heavily on API-accessible models and cloud services as part of their operational fabric. Mitigation efforts and lessons for cybersecurity teams. In response, METR undertook multiple remediation actions: isolating public-facing applications from internal infrastructure, rotating and revoking compromised credentials, shutting down legacy systems, and expanding security staffing. Enhanced monitoring and spend limit alerts were implemented to better detect anomalous API usage and prevent similar large-scale credit consumption in the future. For cybersecurity leaders and analysts, this case serves as a potent example of how fail-open vulnerabilities and insufficient API governance can lead to high-impact financial damage and persistent threat actor presence. It underscores the need for: * Rigorous API authentication and authorization controls * Comprehensive monitoring including spend and usage anomaly detection * Segmentation of public and internal environments * Proactive credential rotation and secret management As AI research organizations increasingly depend on cloud-based APIs, these lessons are critical to safeguarding intellectual property and controlling operational costs. Conclusion. The METR API key theft and subsequent $600,000 fraudulent credit consumption incident provide a cautionary tale about the cybersecurity risks in AI research infrastructure. Despite METR's security maturity and partnerships with major AI vendors, a single misconfiguration exposed them to prolonged attack and significant financial loss. This breach illustrates how attackers are adept at exploiting overlooked API vulnerabilities and the importance of layered defenses including strict access controls, continuous monitoring, and rapid incident response. Cybersecurity professionals working with AI services must recognize that API security and cloud governance are integral to protecting both data and costly computational resources in this evolving threat environment.

Attacker stole a METR API key, used $600K worth of credits, and no one noticed for weeks. The model provider gave METR the credits for free. An actual customer would not have been so lucky Published Tue 01 Sept 2026 // 20:45 UTC AI model testing organization METR has disclosed two attacks that happened earlier this year, including one in which an attacker stole an API key and spent three weeks consuming public-model credits worth about $600,000. METR (short for Model Evaluation and Threat Research) found no evidence that the attackers accessed sensitive information in either incident, and the org said it investigated both with security experts. METR researchers worked with OpenAI to investigate how its agents hacked Hugging Face, and on Monday, it disclosed two of its own security snafus. "In March 2026, attackers stole an API key for inference on public models and consumed a substantial amount of credits," the nonprofit disclosed in a Monday report. "In May 2026, we observed attackers systematically probing our publicly accessible infrastructure, including an unsuccessful attempt to access internal data via an inadvertently exposed endpoint." From fail-open bug to model-credit theft. The March incident involved a METR researcher who didn't have access to sensitive information - including model data and credentials, as well as information about model architectures, training, and release dates. The researcher used agents running on a personal EC2 instance that was "intentionally" left publicly accessible behind Google authentication. The instance contained an API key for METR's public models account. According to METR's account, a "vibe-coded app" included a fail-open bug that disabled authentication, and this exposed the system to the public internet for several days. "We suspect that the attacker found the instance by looking through recently-registered websites (e.g. in certificate transparency lists) to find vibe-coded sites with high-signal keywords relating to LLMs or agents, for purposes of harvesting potentially exposed model provider API keys," the AI research org wrote. Once the attacker found the app, they prompted an agent to reveal its model provider API key, then added an SSH key to maintain persistent access, and over the next three weeks used the stolen credentials to consume API credits on public models worth about $600,000. Luckily for METR, the unnamed model developer had given the credits to the nonprofit for free. How do you not notice the 'large illicit usage?' METR does answer the question on everyone's mind in the report: Why its researchers didn't notice the "large illicit usage?" There are several reasons for this. First, the model testing operation regularly runs evaluations that use a lot of tokens, and this means the organization is "very acclimated to getting lots of weird rate limit and API errors." So the high usage didn't look that out of the ordinary. Plus, since the tokens were free, METR didn't accrue a large bill, and at the time there was no way to put a spending limit on keys like the one that was stolen. In response to the March incident, METR says it improved its security infrastructure, protocols, and review process, and will continue to invest in security. To this end, it also hired a security lead, and plans to add more security staff. Crims used agents to try to access frontier models. The second incident happened in early May, when "METR became the target of a sustained external attack campaign." After being "tipped off" that attackers who appeared financially motivated may have been trying to gain illicit access to frontier models, METR watched the intruders probe its publicly accessible infrastructure. They also used agents to find ways to gain initial access, including automated vulnerability discovery, credential stuffing against authentication providers, attempting OAuth token grants, scanning newly deployed services, and phishing attempts. At the same time, METR unintentionally "exposed a read-only SQL query mechanism via our public transcript viewer." While queries were scoped to public data by default, a bug allowed access to unpublished evaluation data, and "some sensitive model data was accidentally included in this database." However, there's no evidence that the attacker found the exploit or accessed any non-public data, according to the model testing body. An independent bug hunter discovered the vulnerability and reported it to METR, which paid the researcher a bounty, and took the API offline. In response, METR says it now uses an isolated production environment for public-facing applications that is separate from its internal infrastructure.(R)