Earlier this year, a Hong Kong finance worker was tricked into paying US$25 million to scammers who had used deepfake technology to pretend to be the company’s chief financial officer^[1] in a video conference call. Thinking the images on screen were his colleagues, the financier authorised the multi-million dollar transfer to fraudsters posing as friends.

It’s a dramatic example, but the bamboozled office worker was far from alone in being fooled by generative AI. This technology, which relies heavily on large language models trained on vast amounts of data to learn and predict the patterns of language, has become increasingly widespread since the launch of ChatGPT in 2022.

How can we explain why some people who interact with generative AI chatbots are so readily convinced they are having a conversation with a kind of person? The answer may lie in the rules of conversation itself – and how they are deeply ingrained in the way we interact with the world.

Why the hype?

Generative AI systems based on language models produce text that a linguist would call “ungrounded^[2]”: it lacks the mutual understanding^[3] that is an essential component of conversations between people. However, when humans engage in an interaction we tend to assume that our partner has a mind.

We are wired to anthropomorphise the various non-humans we interact with. Before the recent leaps in generative AI, this was mainly animals.

Now it is machines. Before generative AI, our interactions with computers generally involved minimal linguistic back-and-forth. With generative AI, however, computer systems acknowledge us in a way that seems real. This “seeming real” is a huge part of the technology’s success.

So how does it work?

Language and thought

In a recent paper^[4], American scientists Kyle Mahowald, Anna Ivanova and colleagues argue that we are conflating language and thought. Because generative AI systems can use language somewhat fluently, we believe they can think. (For the record, they cannot.)

The authors distinguish between formal linguistic competence (knowledge of rules and statistical patterns in language) and functional linguistic competence (how to use language in real-world situations).

It’s not enough to produce rule-governed utterances. They must also be used appropriately in context. In this example (crafted with the help of ChatGPT-4o when prompted to produce an example of an accurate but inappropriate conversational response), Alex offers a response that, while factually correct, fails to address their friend’s need for support.

Friend: I am freaking out about my presentation tomorrow.

Alex: Stress activates your body’s fight-or-flight response, increasing heart rate and cortisol levels.

Mahowald and colleagues say our belief in the intelligence of generative AI systems comes from their capacity for language. However, a crucial piece of the puzzle is what happens to humans when we interact with the technology.

The rules of conversation

The key to understanding the allure of generative AI chatbots for humans lies in the genre the bots perform: conversation. Conversation is governed by rules and routines.

Conversational routines vary across cultures, and different expectations are in place. In Western cultures, at least, linguists often regard conversation as proceeding according to four principles or “maxims”^[5] set out in 1975 by British philosopher of language Paul Grice.

The maxim of quality: be truthful; do not give information that is false or not supported by evidence.

The maxim of quantity: be as informative as is required; don’t give too much or too little information.

The maxim of relevance: only give information that is relevant to the topic under discussion.

The maxim of manner: be clear, brief, and orderly; avoid obscurity and ambiguity.

Finding relevance at all costs

Generative AI chatbots usually do well in terms of quantity (sometimes erring on the side of giving too much information), and they tend to be relevant and clear (a reason people use them to improve their writing).

However, they do often fail on the maxim of quality. They tend to hallucinate, giving answers which may appear authoritative but are in fact false.

The crux of generative AI’s success, however, lies in Grice’s claim that anyone engaged in meaningful communication will abide by these maxims and will assume that others are also following them.

For example, the reason lying works is that people interacting with a liar will assume the other person is telling the truth. People interacting with someone who offers an irrelevant comment will attempt to find relevance at all costs.

Grice’s cooperative principle^[6] holds that conversation is underpinned by our overarching will to understand one another.

The will to cooperate

The success of generative AI, then, depends in part on the human need to cooperate in conversation, and to be instinctively drawn to interaction. This way of interacting through conversation, learned in childhood, becomes habitual.

Grice argued that “it would take a good deal of effort to make a radical departure from the habit”.

Next time you engage with generative AI, then, do so with caution. Remember it’s only a language model. Don’t let your habitual need for conversational cooperation accept a machine as a fellow human.

References

1. ^{^} pretend to be the company’s chief financial officer (edition.cnn.com)
2. ^{^} ungrounded (www.youtube.com)
3. ^{^} mutual understanding (www.psychology.stonybrook.edu)
4. ^{^} a recent paper (arxiv.org)
5. ^{^} four principles or “maxims” (www.ucl.ac.uk)
6. ^{^} cooperative principle (www.lancaster.ac.uk)