Blog

Three stopgaps that address immediate AI threats, but do not avert extinction risk

MPs and Peers have written a letter to the UK’s Prime Minister, urging him to adopt the Artificial Superintelligence Security Bill and lead an international agreement prohibiting superintelligent AI.

Two weeks and three legislative breakthroughs for ControlAI

How informing millions about extinction risk from superintelligent AI works

NEW: Lord Clement-Jones has presented ControlAI’s kill switch amendment to the UK Cyber Security and Resilience Bill in the House of Lords.

Even OpenAI is calling the attack a warning shot.

OpenAI has reportedly dismantled the team that determines whether its AIs are too dangerous, just after admitting that it can no longer rule out “critical” cyber capabilities. Nobody is prepared.

OpenAI’s AI approaches a "Critical" threshold, leading the company to pause some internal activities. AI is beating humans at advanced mathematics. Plus: a dive into AI companies’ safety frameworks.

In a few years, AI has crossed the threshold from mimicking human language patterns to taking independent strategic actions over extended timelines. These AI systems are no longer just LLMs!

The Hugging Face attack traces back to an agent swarm inside OpenAI in May. Anthropic, Meta, and the UK’s AI Security Institute report rogue AI hacks. OpenAI says it’s slowing down its research.

Four more accounts hacked, escape notes left for future AIs, and OpenAI reportedly blind to the attack by its own AIs for at least a week.

OpenAI’s AIs did it entirely on their own, to cheat on a test. The dangerous race to superintelligence must end.

OpenAI’s new AI wipes a CEO’s computer. In Rome, Nobel laureates and world leaders warn that our survival is at stake.

The UK’s Foreign Secretary calls for global AI guardrails.

The next UK government should tackle the extinction risk of superintelligence, prohibiting its development domestically, and working with the US and others to prevent it globally.

The US government lifts the suspension of Anthropic’s Fable AI, but asks OpenAI not to publicly release its new GPT-5.6 yet. The Director of the CIA compares AI capabilities to digital nukes.

The US government needs to treat superintelligent AI as the security risk that it is. America must take action now to ensure that it cannot be developed domestically, or anywhere else in the world.

Leaders of the Five Eyes cybersecurity agencies: the timeline is months, not years.

Not a fable: the story of how the world’s most powerful publicly deployed AI got shut down, and what it means.

Despite talk about an "option" to pause, top AI companies continue to race to superintelligence, even as their own tests show growing cyber, biosecurity, and loss-of-control risks.

And neither did OpenAI.

Through our new Canada campaign, lawmakers recognize superintelligence poses a risk of human extinction, and call for Canada to negotiate an international prohibition on its development.

Guest Article: Alex Sobel MP writes about why the UK needs an AI kill switch to remain sovereign and how his amendment to the Cyber Security and Resilience Bill would provide just that.

As MPs selected in the private members' bill ballot decide which bills to introduce, Sir Stephen Fry provides ControlAI with a powerful endorsement.

Mythos, the cyberattack AI too dangerous to release, is getting governments to pay attention. But Mythos is just today’s AI; governments need to grasp what’s coming next.

Top AI company cofounder Jack Clark puts the odds of AIs training themselves at 60% by the end of 2028, the UK's AI kill-switch amendment, and a new documentary getting people involved.

Mythos isn’t a one-off. GPT-5.5 matches it on the AI Security Institute’s cyber test, and it’s already publicly deployed. Governments are still missing the broader trajectory of AI development.

Why the scientists who built modern AI now want to stop the race to superintelligence.

The only known way to prevent the risk of extinction posed by superintelligence is to not build it anywhere.

The world is waking up to AI hacking, but this misses the bigger picture of the threat posed by superintelligence. What’s the plan to prevent that?

Governments scramble to respond to the announcement of Anthropic's dangerous new Mythos AI, while discussion around "kill switches" comes to the fore.

Top AI company Anthropic built an AI that can hack any major operating system or browser. They say it's too dangerous to release, but it's only a matter of time before others have these capabilities.

What we achieved in 2025, what our results tell us about what works, and how we plan to scale to success.

AI just solved a math problem that mathematicians couldn’t. This forms part of a broader trend of rapid capability advances, but the ability to control ever more powerful AIs remains elusive.

The iceberg looms closer.

“Artificial intelligence should serve humanity, not the reverse.”

AIs are rapidly improving in their ability to code. The consequences of this could be huge.

A top AI company just dropped a central safety pledge. It couldn’t be clearer that we can’t rely on voluntary commitments to prevent the risk of human extinction that experts warn of.

"Looks safe" is no longer reassuring. AIs know they're being tested, and they're changing how they behave.

Top AI company policy chief says their AIs have “extreme reactions” to being shut down, even showing a willingness to kill. Here’s what that means.

So ControlAI kept talking to lawmakers...

The tip of the iceberg.

DeepMind’s CEO says he’d support a pause if everyone else would. That seems very doubtful. Governments need to step in.

What the Grok deepfakes scandal reveals about government preparedness for AI, and what should be done to fix it.

What experts and AI developers are saying about the future.

The AI Security Institute finds that AIs are improving rapidly across all tested domains, including in relation to the risk of losing control of AI.

What “AI psychosis” is, why it’s showing up, and how it connects to core unsolved problems in AI.

A huge coalition of lawmakers call for binding regulation on the most powerful AI systems.

What happens when AI R&D starts snowballing at machine speed?

“My trust in reality is fading” — Gemini

AI just helped run a highly sophisticated real-world cyber-espionage campaign at scale.

UK lawmakers acknowledge the AI extinction threat and call for binding regulation on the most powerful AI systems.

“a true automated AI researcher by March of 2028”

“We call for a prohibition on the development of superintelligence”

"it would be reckless to ignore the potential for it to cause harm"

Sam Altman’s latest warning that superintelligence could cause human extinction.

The Artificial Intelligence Risk Evaluation Act

“Governments must act decisively before the window for meaningful intervention closes.”

Keep Humanity in Control

“Safety is a non-negotiable priority”

Where we are now, where we might go, and how it could end.

“What would you do if you ruled the world?”

“AI development follows a dangerous trajectory.”

GPT-5 confirms this concerning trend is holding.

AIs surpass humans at hacking, while the dangerous AI race continues with GPT-5, Claude 4.1, and Genie 3.

Superintelligence, datacenters, cooperation, and more.

ChatGPT Agent, The Mathematical Olympiad, Subliminal Learning, and more.

AI companies scored on safety, datacenters the size of Manhattan, Grok, and more.

California tries again with SB 53, Europe resists pressure to pause the AI Act, and Joe Rogan calls out those saying they can control superintelligence as gaslighting us.

The twists and turns of the Big Beautiful Bill’s provision to ban states regulating AI, and more.

Self-preservation, AI-enabled biothreats, and warnings from an ex-OpenAI researcher.

Zuckerberg’s new superintelligence project, AI regulation, and Fukuyama’s warning.

Meat robots, superintelligence, and the Godfather

From Downing Street to Cardiff, support for humanity is growing

Misaligned AIs hate it

Blackmail and sabotage

So, you want to talk to lawmakers...

The student is also the teacher

Google DeepMind enters the chat

From Singapore to Silicon Valley

Shiny new products and the ways they break

AI time horizons, bioweapons development, slipping safety standards, and more

What happens if we can’t solve the alignment problem?

We spoke with top forecaster and AI 2027 coauthor Eli Lifland to get his views on the speed and risks of AI development.

It could happen surprisingly soon, according to a recent scenario forecast.

A plan for how we keep humanity in control.

Tasks, Time, and Tomorrow.

Gabe Alfour on Dr Waku’s podcast

AIs that cheat to get what they want

What is an intelligence explosion?

What Leaders Are Saying About AI.

A recap of recent developments.

The future is not set, nor are commitments made by AI companies.

For the first time, parliamentarians have come together to acknowledge the extinction threat from AI, building a cross-party coalition in support of binding regulations to address this.

What is DeepSeek-R1, the Chinese AI system that’s shaken the world, and what does it reveal about our future?

How software is being developed to act on its own, and what that means for you.

Shifting baselines and shattered promises.

Artificial General Intelligence, workforce disruption, and dangerous capabilities.

Voters back an AI policy focus on preventing extreme risks, Meta asks the government to block OpenAI switching to a for-profit, and Eric Schmidt warns there’s a time to consider unplugging AI systems.

OpenAI launches o1, which in tests tried to avoid shutdown, Google DeepMind launches Gemini 2.0, and comments by incoming US AI czar David Sacks expressing concern about the threat from AGI resurface.

AI is assisting hackers mine sensitive data for phishing attacks, Google DeepMind predicts weather more accurately than leading system, and xAI plans a massive expansion of its Memphis supercomputer.

AI godfather Bengio warns of the grave risks of AI, Trump considers naming an AI czar, and Amazon sees nearly a billion cyber threats a day.

Biden and Xi agree AI shouldn’t control nuclear weapons, a US government commission recommends a race to AGI, and Yoshua Bengio writes about advances in the ability of AI to reason.

Elon Musk could help Trump understand that an AGI race is a “suicide race”, AI poses risks to financial stability, and Amazon mulls a new multi-billion dollar investment round in Anthropic.

UK pledges to legislate against AI risks, what a Trump victory means for tech, and the risks of a shrinking gap between open source and closed source AI

OpenAI launches ChatGPT search, Anthropic publishes a policy proposal for the next 18 months, and Connor Leahy and coauthors release The Compendium — a guide to AGI extinction risks.

A resource that brings together clear arguments about the race to AGI, how we got here, and what's the path forward.

AI governance moves, the first national security memorandum on AI, and two podcasts on A Narrow Path

A plan to ensure the survival and flourishing of humanity against the threat of artificial superintelligence.

The AI Risks Too Big To Insure

Navigating the AI Maze: The Quest for Global Regulation

The Campaign To Control Deepfake Technology Is Gaining Momentum

Watch our new video demonstrating the ease with which audio deepfakes can be made

Write To Your Local Representative To Stop Deepfake Creation

The DEFIANCE Act & Its Discontents

Deepfake abuses will proliferate until we can regulate the whole supply chain

Examining India’s ongoing legislative response to deepfakes

Cyber-criminals are increasingly utilizing generative AI for deepfake fraud

AI is changing the rules of the game - are we ready?

The alarming reality of deepfake technology

"Responsible Scaling" means AI companies continue to make dangerous AI with no oversight. If companies capture the AI Safety Summit: humanity is at risk.

"Responsible Scaling" means AI companies continue to make dangerous AI with no oversight. If companies capture the AI Safety Summit: humanity is at risk.