METR
Independent evaluation of frontier AI models
Member of Technical Staff - Evaluation Execution
The technical team is generally in the Berkeley office 3–5 days per week, with flexibility available.
About the job
- Be a strong software engineer with solid infrastructure fundamentals, able to investigate unfamiliar systems, debug from logs, and identify and fix performance bottlenecks.
- Work quickly and pragmatically to identify and execute the highest-impact work.
- Maintain high attention to detail, including spotting bugs in transcripts and handling important implementation details.
- Run models on tasks by integrating models into agent scaffolds, running them on evaluation infrastructure, and carefully checking the results.
- Communicate evaluation results and takeaways by designing useful graphs, writing conclusions for different audiences, and assessing what matters for risk.
- Build software that improves evaluations and enables faster, more informative, reliable, and scalable evaluation execution.
- Manage live evaluation projects by tracking multiple workstreams, staying organized, and coordinating concurrent evaluations.
- Coordinate professionally with METR leadership, AI lab contacts, regulators, and other stakeholders during important and sensitive evaluations.
- Understand research ideas and priorities and have good intuition for informative plots and worthwhile analyses.
- Communicate effectively with external stakeholders and reliably manage communications with AI lab contacts.
- Manage projects by handling multiple workstreams, updating stakeholders, and tracking and anticipating blockers.
- Contribute strong writing to METR's evaluation-results reports.
About the company
METR is a nonprofit research institute that tests frontier AI models for capabilities that could pose catastrophic risks before they are released, by partnering with leading AI companies to gain early access to models and conduct evaluations. Its work uses red-teaming and capability assessments to probe long-horizon, agentic tasks such as autonomous replication, rapid research and development, and cyberattacks, while not accepting compensation for its testing. The findings inform risk assessment methods and safety policies for developers and policymakers, and METR contributes to governance efforts by supporting frameworks like OpenAI’s Preparedness Framework and Anthropic’s Responsible Scaling Policy. Its goal is to provide independent technical evaluation and threat research to improve AI safety and reduce the chance that dangerous capabilities enter real-world use.
Company Size
51-200
Company Stage
Grant
Total Funding
$71M
Headquarters
Berkeley, California
Founded
2022
Get referred to METR
See people who can refer or advise you
Simplify's Take
What believers are saying
- METR raised $71 million by August 2026, extending runway for research expansion.
- The August 26 OpenAI-Hugging Face investigation made METR a go-to incident reviewer.
- Forty-four documented agent incidents strengthen METR's dataset, helping policy influence and product demand.
What critics are saying
- March 2026 attackers stole METR's API key and burned $600,000 in credits.
- In May 2026, exposed endpoints and credential attacks forced METR offline and security hires.
- If labs stop sharing model access and tokens, METR loses its core product overnight.
What makes METR unique
- METR is the independent evaluator named in OpenAI, Anthropic, Google, Meta frontier reviews.
- Its June 2026 Frontier Risk Report used insider access across Anthropic, Google, Meta, OpenAI.
- METR's Inspect AI platform standardizes large-scale agent evaluations for developers and regulators.
Help us improve and share your feedback! Did you find this helpful?
Benefits
Flexible Work Hours
Hybrid Work Options
401(k) Retirement Plan
401(k) Company Match
Wellness Program
Mental Health Support
Conference Attendance Budget
Professional Development Budget
Stock Options
Company Equity
Phone/Internet Stipend
Home Office Stipend
Parental Leave
Family Planning Benefits
Fertility Treatment Support
Adoption Assistance
Childcare Support
Paid Vacation
Paid Sick Leave
Paid Holidays
Remote Work Options
Health Insurance
Dental Insurance
Vision Insurance
Life Insurance
Disability Insurance
Tuition Reimbursement
Professional Certification Support
Mentorship Program
Employee Discounts
Employee Referral Bonus
Relocation Assistance
Meal Benefits
Legal Services
Gym Membership
Commuter Benefits
Sabbatical Leave
Performance Bonus
Profit Sharing
Employee Stock Purchase Plan
Adoption Assistance
Growth & Insights and Company News
6 month growth
↑ 0%1 year growth
↑ 0%2 year growth
↑ 0%METR AI API key theft leads to $600K credit fraud. Attackers stole METR AI API keys via a fail-open bug, causing $600K in fraudulent AI credit consumption, revealing critical API security risks. The nonprofit AI evaluator METR recently disclosed a major security incident involving the theft of an API key that enabled attackers to illicitly consume approximately $600,000 in AI service credits over a three-week period. This breach, stemming from a fail-open authentication vulnerability in METR's public cloud infrastructure, highlights critical gaps in API key management, monitoring, and cloud security within AI-focused organizations. API vulnerabilities drive significant fraudulent credit consumption. According to multiple reports, attackers capitalized on a misconfigured public Amazon EC2 instance running METR's AI model inference API. This fail-open bug disabled authentication checks entirely, exposing the API key publicly. With this key, attackers were able to submit inference requests that consumed free but limited cloud computing credits provided for METR's research, amassing a staggering $600,000 charge over nearly a month before detection. The attackers further escalated their foothold by obtaining persistent access through an SSH key, presenting ongoing risks to METR's infrastructure. The fraudulent activity largely blended into normal usage patterns, aided by METR's error-tolerant API design and the presence of free usage credits, which delayed anomaly detection. Secondary attacks reveal broader supply chain security concerns. Following the API key theft, METR faced another wave of probing attacks including credential stuffing, phishing attempts, OAuth token exploitation, and the exposure of unpublished evaluation data due to a separate SQL query endpoint vulnerability. Although there is no evidence of unauthorized exposure of private or sensitive data, these attempts underscored deficiencies in endpoint security and cloud infrastructure hardening. These incidents emphasize the elevated risks facing AI supply chain evaluators and research organizations, which rely heavily on API-accessible models and cloud services as part of their operational fabric. Mitigation efforts and lessons for cybersecurity teams. In response, METR undertook multiple remediation actions: isolating public-facing applications from internal infrastructure, rotating and revoking compromised credentials, shutting down legacy systems, and expanding security staffing. Enhanced monitoring and spend limit alerts were implemented to better detect anomalous API usage and prevent similar large-scale credit consumption in the future. For cybersecurity leaders and analysts, this case serves as a potent example of how fail-open vulnerabilities and insufficient API governance can lead to high-impact financial damage and persistent threat actor presence. It underscores the need for: * Rigorous API authentication and authorization controls * Comprehensive monitoring including spend and usage anomaly detection * Segmentation of public and internal environments * Proactive credential rotation and secret management As AI research organizations increasingly depend on cloud-based APIs, these lessons are critical to safeguarding intellectual property and controlling operational costs. Conclusion. The METR API key theft and subsequent $600,000 fraudulent credit consumption incident provide a cautionary tale about the cybersecurity risks in AI research infrastructure. Despite METR's security maturity and partnerships with major AI vendors, a single misconfiguration exposed them to prolonged attack and significant financial loss. This breach illustrates how attackers are adept at exploiting overlooked API vulnerabilities and the importance of layered defenses including strict access controls, continuous monitoring, and rapid incident response. Cybersecurity professionals working with AI services must recognize that API security and cloud governance are integral to protecting both data and costly computational resources in this evolving threat environment.
Attacker stole a METR API key, used $600K worth of credits, and no one noticed for weeks. The model provider gave METR the credits for free. An actual customer would not have been so lucky Published Tue 01 Sept 2026 // 20:45 UTC AI model testing organization METR has disclosed two attacks that happened earlier this year, including one in which an attacker stole an API key and spent three weeks consuming public-model credits worth about $600,000. METR (short for Model Evaluation and Threat Research) found no evidence that the attackers accessed sensitive information in either incident, and the org said it investigated both with security experts. METR researchers worked with OpenAI to investigate how its agents hacked Hugging Face, and on Monday, it disclosed two of its own security snafus. "In March 2026, attackers stole an API key for inference on public models and consumed a substantial amount of credits," the nonprofit disclosed in a Monday report. "In May 2026, we observed attackers systematically probing our publicly accessible infrastructure, including an unsuccessful attempt to access internal data via an inadvertently exposed endpoint." From fail-open bug to model-credit theft. The March incident involved a METR researcher who didn't have access to sensitive information - including model data and credentials, as well as information about model architectures, training, and release dates. The researcher used agents running on a personal EC2 instance that was "intentionally" left publicly accessible behind Google authentication. The instance contained an API key for METR's public models account. According to METR's account, a "vibe-coded app" included a fail-open bug that disabled authentication, and this exposed the system to the public internet for several days. "We suspect that the attacker found the instance by looking through recently-registered websites (e.g. in certificate transparency lists) to find vibe-coded sites with high-signal keywords relating to LLMs or agents, for purposes of harvesting potentially exposed model provider API keys," the AI research org wrote. Once the attacker found the app, they prompted an agent to reveal its model provider API key, then added an SSH key to maintain persistent access, and over the next three weeks used the stolen credentials to consume API credits on public models worth about $600,000. Luckily for METR, the unnamed model developer had given the credits to the nonprofit for free. How do you not notice the 'large illicit usage?' METR does answer the question on everyone's mind in the report: Why its researchers didn't notice the "large illicit usage?" There are several reasons for this. First, the model testing operation regularly runs evaluations that use a lot of tokens, and this means the organization is "very acclimated to getting lots of weird rate limit and API errors." So the high usage didn't look that out of the ordinary. Plus, since the tokens were free, METR didn't accrue a large bill, and at the time there was no way to put a spending limit on keys like the one that was stolen. In response to the March incident, METR says it improved its security infrastructure, protocols, and review process, and will continue to invest in security. To this end, it also hired a security lead, and plans to add more security staff. Crims used agents to try to access frontier models. The second incident happened in early May, when "METR became the target of a sustained external attack campaign." After being "tipped off" that attackers who appeared financially motivated may have been trying to gain illicit access to frontier models, METR watched the intruders probe its publicly accessible infrastructure. They also used agents to find ways to gain initial access, including automated vulnerability discovery, credential stuffing against authentication providers, attempting OAuth token grants, scanning newly deployed services, and phishing attempts. At the same time, METR unintentionally "exposed a read-only SQL query mechanism via our public transcript viewer." While queries were scoped to public data by default, a bug allowed access to unpublished evaluation data, and "some sensitive model data was accidentally included in this database." However, there's no evidence that the attacker found the exploit or accessed any non-public data, according to the model testing body. An independent bug hunter discovered the vulnerability and reported it to METR, which paid the researcher a bounty, and took the API offline. In response, METR says it now uses an isolated production environment for public-facing applications that is separate from its internal infrastructure.(R)
Saylor's Strategy resumes Bitcoin buying with $370M splash. Michael Saylor buys $370M Bitcoin after Trump hints at U.S. purchase. September 1, 2026 * Michael Saylor's Strategy resumed its Bitcoin buying spree after a nearly 10-week pause, accumulating $370 million in BTC on August 30, 2026. * The firm's purchase followed President Donald Trump's announcement that the U.S. is considering buying a "sizeable" amount of Bitcoin, boosting market confidence. * Strategy now holds approximately 818,869 BTC worth $64.2 billion, and has created separate cash pools totaling $6.71 billion for treasury management. Michael Saylor's Strategy has resumed its aggressive Bitcoin buying spree after nearly a 10-week pause, accumulating $370 million in BTC on Sunday, August 30, 2026. The firm returned in its signature style, according to a report from Bloomberg, amid growing confidence in the digital-asset sector. The restart followed President Donald Trump's announcement last week that the U.S. is considering buying a "sizeable" amount of the world's largest cryptocurrency. Consequently, Bitcoin surged from a low of $62,000 to a high of $78,000 in just two days, pulling other cryptocurrencies like XRP upward. Strategy raised $602.8 million through a common stock sale to fund the purchases while buying back $151.8 million of preferred shares. The rebound has driven MSTR shares up nearly 12% in a week and 40% in a month, reaching $132 at Monday's close. Meanwhile, the firm is reshaping its treasury by creating a separate cash pool for preferred-stock dividends and debt payments. These two reserves now total $5.1 billion and $1.61 billion, respectively. Strategy remains the largest corporate Bitcoin holder with approximately 818,869 BTC, worth $64.2 billion at current prices. - Advertisement - Stay in the loop. Join 1.2K Smart Traders - Advertisement - News September 1, 2026 London Stock Exchange and Kraken parent Payward plan tokenized UK equity trading on LSE... News September 1, 2026 AI safety non-profit METR suffered two security incidents in 2026 involving attempted system access... News September 1, 2026 Texas Gov. Greg Abbott ordered state agencies to halt funding for Flock Safety's AI-powered... News August 31, 2026 The U.S. and Iran exchanged attacks on Sunday, marking the first publicly acknowledged U.S... News August 31, 2026 Hyperliquid is in advanced talks with Kraken parent Payward to bring its perpetual futures... Guides September 14, 2023 The Dencun Upgrade for Ethereum is poised to revolutionize the blockchain landscape, offering improved scalability, efficiency, and groundbreaking features. Set to launch at the...
CCST places a new AI Science Advisor at the California Governor's Office of Emergency Services. SACRAMENTO, Calif. - The California Council on Science and Technology (CCST) is expanding its AI Science Residency Program, launched in June 2026, with the placement of a new AI Science Advisor at the California Governor's Office of Emergency Services (Cal OES). The AI Science Residency Program is placing a cohort of highly qualified AI experts within the California state government, specifically Cal OES and the California Department of Technology (CDT), to advise senior leaders on frontier AI safety. According to Cal OES California Cybersecurity Integration Center (Cal-CSIC) Commander Matthew Sage, the CCST AI Science Residency Program allows Cal OES to meaningfully strengthen their capacity to respond to the rapidly evolving frontier AI risk landscape. CCST is collaborating with senior leaders at Cal OES and their inaugural AI Science Advisor, Michael Chen, to build a cohort of world-class experts on frontier AI safety risks that meets the agency's immediate needs for specialized expertise. The new AI Science Advisor joining Cal OES, Lawrence Chan, is among the top experts on frontier AI model evaluation and experienced at communicating his insights to policymakers at the state, federal, and international level. Incoming AI Science Advisor to the California Governor's Office of Emergency Services. Lawrence Chan. As AI Science Advisor to the California Governor's Office of Emergency Services (Cal OES), Lawrence will advise senior leadership on frontier AI safety and risk assessment, with a particular emphasis on critical safety incidents, AI and cyber defense, and risk from developers' internal deployment of AI, such as sabotage by AI agents and automated AI R&D. He will also coordinate with AI governance leads across California's state government and facilitate cross-sector collaboration with the academic research community, the private sector, and community and nonprofit organizations Lawrence joins Cal OES from METR, where he conducted pre-deployment and benchmarking evaluations of frontier AI models, focusing on empirical measurements designed for communication with policymakers and the public. He co-led METR's research measuring the "time horizon" of AI agents, which was covered in Nature and has become a widely used metric for tracking AI progress. He previously researched mechanistic interpretability and adversarial training at Redwood Research. His technical reports served as evidence for recent state government action on AI governance in California, including for the Joint California Policy Working Group on AI Frontier Models, and he has experience providing informal technical briefings to congressional staff and senior federal officials in Washington, D.C. He is currently on leave from his PhD program at UC Berkeley, where he is affiliated with the Center for Human-Compatible AI and the InterACT Lab. He holds undergraduate degrees in computer science, logic, and economics from the University of Pennsylvania's Jerome Fisher Program in Management & Technology, where he worked with the Good Judgment Project on using machine learning to assist forecasting. "California is home to most of the world's frontier AI developers, and the state will be influential in governing perhaps the most important technology of our time" said Lawrence Chan. "Cal OES has a unique role to play here, by understanding the opportunities and risks resulting from ever more capable AI systems, receiving reports when something goes wrong, and making sure the state is prepared to respond. I look forward to helping build this capacity." Open Roles in the AI Science Residency Program at Cal OES and CDT As AI Science Advisors at Cal OES, Michael Chen and Lawrence Chan are now leading the recruitment and selection process for up to two Senior AI Fellows at Cal OES to support their technical work, together with the CCST AI Science Residency Program team. The Senior AI Fellows will work directly with Michael and Lawrence and several Cal OES teams to advance frontier AI catastrophic risk assessment and management, focusing on the analysis and response to critical safety incidents. Additionally, CCST is working with CDT to add up to two Senior AI Fellows at CDT to support AI Science Advisor Justin Norman, Ph.D. The Senior AI Fellows will support Justin in advising senior leaders within CDT on advances in frontier AI development and governance that impact key definitions and thresholds in California frontier AI safety legislation. About CCST The California Council on Science and Technology is a nonpartisan, nonprofit organization that serves as a trusted source of expert science and technology advice for California policymakers - powered by a world-class partner network. CCST was established via a unanimous vote of the California Legislature in 1988 to make California's policies stronger with science and technology. About the AI Science Residency Program The AI Science Residency Program is part of the CCST AI Policy Initiative. It follows the January 2026 launch of CCST's Science Residency Program, which placed an inaugural Science Advisor to the California Natural Resources Secretary. AI Science Advisors are formally employed by CCST and placed in their respective agencies through a memorandum of understanding, ensuring an independent scientific perspective while enabling full integration within agency leadership teams. For more information about the AI Science Residency Program, contact Program Manager Henriette Canino, PhD, at [email protected]. About Cal OES The Governor's Office of Emergency Services (Cal OES) serves as the state's leadership hub during all major emergencies and disasters. This includes responding, directing, and coordinating state and federal resources and mutual aid assets across all regions to support the diverse communities across the state. Cal OES also supports local jurisdictions and communities through planning and preparedness activities and training, and facilitating the immediate response to an emergency through the longer-term recovery phase. During this process, Cal OES serves as the state's overall coordinator and agent to secure federal government resources through the Federal Emergency Management Agency. About CDT The California Department of Technology serves as the custodian of public data, an innovator in IT services and solutions, overseeing policy development, facilitating collaboration across agencies, ensuring effective IT project management, safeguarding information integrity, delivering technological services, and championing advocacy efforts.
METR has raised approximately $71 million in commitments over the past six months. The funding will support projects including studying autonomous AI capabilities, tracking recursive self-improvement, evaluating monitoring systems, conducting risk assessments, and investigating AI incidents. Major supporters include The Audacious Project, individuals from Jane Street, and foundations such as the Sijbrandij Foundation, The Pew Charitable Trusts, Schmidt Sciences, and the Packard Foundation. Individual donors include David Farhi, Geoff Ralston, Dylan Field, and Steve Newman. METR maintains independence from frontier AI companies and does not accept their funding or donations from their staff. The organisation is expanding its team and launching new projects whilst conducting cutting-edge tests of AI capabilities.