‘Godfather of AI’ now fears it’s unsafe. He has a plan to rein it in

This week the US Federal Bureau of Investigation revealed two men suspected^[1] of bombing a fertility clinic in California last month allegedly used artificial intelligence (AI) to obtain bomb-making instructions. The FBI did not disclose the name of the AI program in question.

This brings into sharp focus the urgent need to make AI safer. Currently we are living in the “wild west” era of AI, where companies are fiercely competing to develop the fastest and most entertaining AI systems. Each company wants to outdo competitors and claim the top spot. This intense competition often leads to intentional or unintentional shortcuts^[2] – especially when it comes to safety.

Coincidentally, at around the same time of the FBI’s revelation, one of the godfathers of modern AI, Canadian computer science professor Yoshua Bengio, launched a new nonprofit organisation^[3] dedicated to developing a new AI model specifically designed to be safer than other AI models – and target those that cause social harm.

So what is Bengio’s new AI model? And will it actually protect the world from AI-faciliated harm?

An ‘honest’ AI

In 2018, Bengio, alongside his colleagues Yann LeCun and Geoffrey Hinton, won the Turing Award for groundbreaking research they had published three years earlier on deep learning^[4]. A branch of machine learning, deep learning attempts to mimic the processes of the human brain by using artificial neural networks to learn from computational data and make predictions.

Bengio’s new nonprofit organisation, LawZero^[5], is developing “Scientist AI”. Bengio has said^[6] this model will be “honest and not deceptive”, and incorporate safety-by-design principles.

According to a preprint paper^[7] released online earlier this year, Scientist AI will differ from current AI systems in two key ways.

First, it can assess and communicate its confidence level in its answers, helping to reduce the problem of AI giving overly confident and incorrect responses.

Second, it can explain its reasoning to humans, allowing its conclusions to be evaluated and tested for accuracy.

Interestingly, older AI systems had this feature^[8]. But in the rush for speed and new approaches, many modern AI models^[9] can’t explain their decisions. Their developers have sacrificed explainability for speed.

Bengio also intends “Scientist AI” to act as a guardrail against unsafe AI. It could monitor other, less reliable and harmful AI systems — essentially fighting fire with fire.

This may be the only viable solution to improve AI safety. Humans cannot properly monitor systems such as ChatGPT, which handle over a billion queries daily. Only another AI can manage this scale.

Using an AI system against other AI systems is not just a sci-fi concept – it’s a common practice in research to compare and test different level of intelligence in AI systems^[10].

Adding a ‘world model’

Large language models and machine learning are just small parts of today’s AI landscape.

Another key addition Bengio’s team are adding to Scientist AI is the “world model^[11]” which brings certainty and explainability. Just as humans make decisions based on their understanding of the world, AI needs a similar model to function effectively.

The absence of a world model in current AI models is clear.

One well-known example is the “hand problem^[12]”: most of today’s AI models can imitate the appearance of hands but cannot replicate natural hand movements, because they lack an understanding of the physics — a world model — behind them.

Another example is how models such as ChatGPT struggle with chess, failing to win and even making illegal moves^[13].

This is despite simpler AI systems, which do contain a model of the “world” of chess, beating even the best human players^[14].

These issues stem from the lack of a foundational world model in these systems, which are not inherently designed to model the dynamics of the real world^[15].