More scientific papers being written with help of ChatGPT—especially in computer science

Since its release in November of 2022, the use of ChatGPT and other large language models (LLMs) has proliferated throughout many disciplines, providing writing assistance for everything from speeches to contracts. So, it may not be surprising that some scientists might utilize ChatGPT to quicken the pace at which they publish their research.

There is little known about how the adoption of AI-generated content might affect the diversity, quality and reliability of research papers. And because these technologies are still new and constantly evolving, there is not yet a sure-fire way to detect the use of LLMs, and many institutions are still developing policies to curb their use.

To get a better grasp of how ChatGPT has been used in scientific writing over the last few years, a group of researchers recently conducted a study analyzing 1,121,912 scientific papers and preprints from arXiv, bioRxiv and Nature portfolio journals. The study, published in Nature Human Behaviour, used a new population-level framework based on word frequency shifts to estimate the increase of LLM-modified content between January 2020 and September 2024.

The study found that abstracts and introductions were most commonly affected, while methods and experiment sections showed less AI use, likely due to the summarization abilities of LLMs. A steady increase in the likely use of ChatGPT was observed across multiple topics of study, with the most dramatic being computer science—a notably AI-adjacent discipline.

The analysis showed likely LLM use in 22.5% of computer science abstracts and 19.5% of computer science introductions by September 2024. In November 2022, these numbers were only around 2.4% and similar across all article types at the time. LLM use was also relatively high in electrical engineering and systems science by 2024, at 18.0% for abstracts and 18.4% for introductions.

LLM usage was found to be much lower in areas like mathematics, with LLM usage at 7.7% for abstracts and 4.1% for introductions. The Nature portfolio of journals also showed a lower increase in AI use, with 8.9% for abstracts and 9.4% for introductions.

In addition to the field of study, the analysis was further stratified by author preprint frequency, paper length, and geographical region, in which the researchers found LLM modification to be more common in a few different cases. Authors who posted preprints more frequently were associated with more LLM usage in their papers, possibly due to increased pressure to put out more papers at a faster pace. Shorter papers—those less than 5,000 words—were also associated with more help from LLMs, along with those in more competitive research areas, like computer science.

Detecting AI-generated text in non-English speaking geographical regions is trickier, and some bias has been pointed out in previous methods of AI detection against non-native English writers in scientific papers. This study did show higher LLM usage in papers from China and Continental Europe, compared to North America and the UK, but much of this is likely for English-language assistance.

As the AI landscape rapidly evolves in the coming years, it has the potential to change how science is written and communicated, which in turn raises questions about transparency, originality and the future of scientific publishing.

The study authors point out many of the questions that should be answered as science continues to incorporate these technologies: “Our observations of the rise of generated or modified papers open many questions for future research. How do such papers compare in terms of accuracy, creativity or diversity? How do readers react to LLM-generated abstracts and introductions? How do citation patterns of LLM-generated papers compare with other papers in similar fields? How might the dominance of a limited number of for-profit organizations in the LLM industry affect the independence of scientific output?

“We hope our results and methodology inspire further studies of widespread LLM modified text and conversations about how to promote transparent, diverse and high-quality scientific publishing.”

Written for you by our author Krystal Kasal, edited by Gaby Clark, and fact-checked and reviewed by Robert Egan—this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive.
If this reporting matters to you,
please consider a donation (especially monthly).
You’ll get an ad-free account as a thank-you.

© 2025 Science X Network