Are we going to kill each other due to AI?

In the last few years, there has been a lot of discussion around risks posed by LLMs and hypothetical future superhuman AIs. This post isn’t about the risks themselves, but about the public debate we’re witnessing concerning them.

Unfortunately, it turned into a highly polarized two-sided debate, where each side seems to be focused on proving they were right all along, rather than engaging in a constructive discussion.

Let’s look at a couple of examples, just to show the above point.

The first one is about the limits of the transformer architecture that is the heart of LLMs. On one side, we have the skeptics saying that no technology can improve forever, and that we’re indeed close to reaching a plateau where, barring some discovery that revolutionizes the architecture we use for LLMs, we will be unable to make new LLMs that are significantly smarter than the previous ones. On the other side, we have the enthusiasts saying that we haven’t seen any sign of slowdowns so far, and so surely LLMs will keep improving until we reach a point where they’re better than any human at every conceivable task. We actually don’t have any scientific theory telling us where the limit of the transformer architecture is (if we do have one, please point it to me!), yet both sides behave as if they’re clearly right, and it is the other side that doesn’t want to see the truth for psychological, economic, or other reasons.

The second example is about the possibility of simulating a human brain with a computer. Some people say that what happens in our brains is just a matter of chemistry; have software simulate those chemical reactions, and you have a human brain running inside a computer. Other people say that there are quantum effects happening in our brain that have an important role in our consciousness and in our subjective perception of the world. Therefore, a classical (thus deterministic) system like a computer cannot simulate it. This summer I saw a video where a famous programmer accused a famous proponent of the “quantum brain theory”, together with all the people sharing his opinion, of being anti-scientific and negationist. This “quantum brain theorist” published some books where he backed his opinion with self-consistent arguments, but these arguments indeed lacked scientific validation. However, the theory presented by the programmer as the absolute and evident truth (that we will soon have such a simulation) lacks scientific validation as well. We only have two ways of proving that it is possible to simulate a human brain. The first and, despite still being very difficult, probably the easiest, is to build such a simulation and show that it is indeed simulating a human brain. The second is providing a formal proof that intelligence, consciousness, feelings, etc. are computable¹ (good luck doing this; we don’t even have formal definitions for these). On the other side, the only way of affirming the impossibility of simulating a human brain is a formal proof that those things aren’t computable (again, with the same difficulties as trying to prove they are).

How can we pretend we’re doing science when both sides present their unsubstantiated opinions as evident truths? We’re turning what should be a science and policy debate into a religious war.

I believe we can, and should, do much better than this. Instead of each side digging in their heels, we should first concede that, after all, both sides have made failed predictions (e.g., LLMs didn’t reach a plateau, and radiologists are still in demand). Then, recognizing that the future is actually uncertain (at the moment, we lack a scientific framework to make well-grounded predictions), we should try to find common ground to discuss what’s best to do in the face of uncertainty. I’ll come clear and say that I believe global warming to be a much more concrete threat to humanity than LLMs. Still, let’s also remember that, when doing threat analysis, risk is defined as “probability times impact”. This means that something that has a low probability of happening, but has a big impact if it happens, is still worth mitigating against. Indeed, every time I’m in a car or on a plane, I wear a seatbelt, despite not expecting to be involved in an accident.

To find such common ground, I suggest we stop arguing about what the future will look like; we can’t back our hypotheses in any meaningful way anyway. Instead, we should look at the current capabilities of AI agents and mitigate threats as new capabilities emerge. On one side, this would mean much less arguing, as the proof of the capabilities will be based on direct observation, rather than on speculation about what will or will not be possible in the future. On the other hand, by gradually mitigating threats as they emerge, we’re also hampering a hypothetical future AI agent deciding to go for world domination.

AI agents are already posing risks now, and no side is denying this. Let’s stop arguing about the future and work together to mitigate the issues we have now. We will then face future threats as they gradually emerge. If we keep fighting, the risk is that we end up not doing anything, which would be the worst possible outcome.

1. Here I’m referring to Computability Theory. I’m writing this clarification because, when I wrote a similar argument on HN some years ago, another commenter replied that surely intelligence is computable, because it is “computed by the universe”. When we say “computable” without further specifications, we mean “computable by a Turing machine”. Our computers are an approximation of a Turing machine, if something isn’t computable by a Turing machine then can’t be computed by our computers as well. We indeed know that there are some aspects of our physical reality that aren’t computable, and therefore cannot be simulated, despite them being somehow “computed by the universe”. [↩]