Is superintelligence dangerous? The main arguments on both sides

The case for serious concern

In 2023 hundreds of AI researchers and industry leaders, among them Geoffrey Hinton, Yoshua Bengio and the heads of several major labs, signed a one-sentence statement saying that reducing the risk of extinction from AI should be a global priority, alongside pandemics and nuclear war.

The main arguments behind that concern:

• Goals and intelligence are separate. Being very smart does not automatically mean having good values. A highly capable system could pursue goals that conflict with ours.
• Useful sub-goals. Almost any goal is easier to reach with more resources and without being switched off. A capable system might therefore resist correction, even if nobody built it to.
• Hard to check. We cannot yet reliably look inside large AI systems to see what they are really optimising for. See our guide to alignment.
• Racing pressure. Companies and countries compete, which rewards speed over caution.

The strongest version of this view comes from Eliezer Yudkowsky and Nate Soares, whose 2025 book If Anyone Builds It, Everyone Dies argues that building superintelligence with anything like today’s methods would very likely end in human extinction. Hinton has given a lower but still serious estimate, saying there is perhaps a 10 to 20 percent chance that AI leads to human extinction within the next few decades.

The case that the risk is overstated

• Today’s AI is far from it. Researchers such as Yann LeCun argue that current language models lack key abilities, like understanding the physical world or planning, and that superintelligence is not around the corner.
• We design the goals. Sceptics argue that AI systems do what they are built and trained to do, and that safety measures can be built in step by step, as with other technologies.
• Real harms now. Critics such as Gary Marcus and many AI ethics researchers argue that focusing on extinction distracts from present problems like misinformation, bias, job loss and concentration of power.
• Hype. Some point out that warnings about world-changing AI also help companies attract investment and attention.

The middle ground

Many researchers sit between these positions. They think catastrophic outcomes are unlikely but possible, and that the cost of preparing is small compared to the cost of being wrong. That view supports more safety research, testing of models before release and international agreements, without assuming disaster is certain.

How this connects to the tracker

The SafeASI forecast assumes no catastrophe happens first, because a market only pays out if someone is around to resolve it. The methodology page explains why that matters when reading the numbers.