Fri Dec 19

What the hyperproduction of AI slop is doing to science

Written by: Vitomir Kovanovic, Associate Professor and Associate Director of the Centre for Change and Complexity in Learning (C3L), Education Futures, University of South Australia

Over the past three years, generative artificial intelligence (AI) has had a profound impact on society. AI’s impact on human writing ^[1], in particular, has been enormous.

The large language models that power AI tools such as ChatGPT are trained on a wide variety of textual data, and they can now produce complex and high-quality texts of their own.

Most importantly, the widespread use of AI tools has resulted in hyperproduction of so-called “AI slop”: low-quality AI-generated outputs produced with minimal or even no human effort.

Much has been said about what AI writing means for education, work, and culture. But what about science? Does AI improve academic writing, or does it merely produce “scientific AI slop”?

According to a new study ^[2] by researchers from UC Berkeley and Cornell University, published in Science, the slop is winning.

Generative AI boosts academic productivity

The researchers analysed abstracts from more than a million preprint articles (publicly available articles yet to undergo peer review) released between 2018 and 2024.

They examined whether use of AI is linked to higher academic productivity, manuscript quality and use of more diverse literature.

The number of preprints an author produced was a measure of their productivity, while eventual publication in a journal was a measure of an article’s quality.

The study found that when an author started using AI, the number of preprints they produced increased dramatically. Depending on the preprint platform, the overall number of articles an author published per month after adopting AI increased between 36.2% and 59.8%.

The increase was biggest among non-native English speakers, and especially for Asian authors, where it ranged from 43% to 89.3%. For authors from English-speaking institutions and with “Caucasian” names, the increase was more modest, in the range of 23.7% to 46.2%.

These results suggest AI was often used by non-native speakers to improve their written English.

What about the article quality?

The study found articles written with AI used more complex language on average than those written without AI.

However, among articles written without AI, ones that used more complex language were more likely to be published.

This suggests that more complex and high-quality writing is perceived as having greater scientific merit.

However, when it comes to articles written with AI support, this relationship was reversed – the more complex the language, the less likely the article was to be published. This suggests that AI-generated complex language was used to hide the low quality of the scholarly work.

AI increased the variety of academic sources

The study also looked at the differences in article downloads originating from Google and Microsoft search platforms.

Microsoft’s Bing search engine introduced an AI-powered Bing Chat feature in February 2023. This allowed the researchers to compare what kind of articles were recommended by AI-enhanced search versus regular search engine.

Interestingly, Bing users were exposed to a greater variety of sources than Google users, and also to more recent publications. This is likely caused by a technique used by Bing Chat called retrieval-augmented generation, which combines search results with AI prompting.

In any case, fears that AI search would be “stuck” recommending old, widely used sources was not justified.

Moving forward

AI has had significant impact on scientific writing and academic publishing. It has become an integral part of academic writing for many scientists, especially for non-native speakers and it is here to stay.

As AI is becoming embedded in many applications such as word processors, email apps, and spreadsheets, it will be soon impossible not to use AI whether we like it or not.

Most importantly for science, AI is challenging the use of complex high-quality language as the indicator of scholarly merit. Quick screening and evaluation of articles based on language quality is increasingly unreliable and better methods are urgently needed.

As complex language is increasingly used to cover up weak scholarly contributions, critical and in-depth evaluations of study methodologies and contributions during peer review are essential.

One approach is to “fight fire with fire” and use AI review tools, such as the one recently published by Andrew Ng at Stanford ^[3]. Given the ever-growing number of manuscript submissions and already high workload of academic journal editors, such approaches might be the only viable option.