Large Language Models Memorise Their Training Data

A new paper shows that LLMs memorise much more of their training data than was previously known.

The idea that generative AI models don’t memorise their training data is one that you hear again and again. It features in AI companies’ efforts to lobby governments to weaken copyright law; it appears in their legal defenses; sometimes, it even makes it into judges’ verdicts.

But it is false.

A new academic paper makes this abundantly clear. It turns out that, by fine-tuning some of the leading large language models, you can get them to regurgitate lots of material from copyrighted books - much more than has been demonstrated before.

This totally undermines AI companies’ arguments that their models don’t memorise what they are trained on, and could have major implications for the various lawsuits that have been brought against them.

Background

We already know that generative AI models trained on copyrighted works can, in some instances, output those works.

Most recently, this was shown in a paper from researchers at Stanford. They showed that portions of copyrighted works could be extracted from four large language models: GPT-4.1, Gemini 2.5 Pro, Grok 3, and Claude 3.7 Sonnet. Like many other examples, they proved this by giving the models passages of copyrighted text and asking them to continue those passages.

But the idea that AI models don’t store their training data continues to abound. This is partly due to AI industry lobbying, and partly because the guardrails that AI companies do implement to limit regurgitation in user-facing AI products - despite being woefully lacking in many cases - can lead people to the erroneous conclusion that memorisation does not occur.

In this context, further evidence of memorisation is incredibly useful.

The new paper: Fine-tuning activating verbatim recall

In this paper, called Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models, the researchers take a new approach. They take three large language models provided by OpenAI, Google, and DeepSeek, and they fine-tune them.

Fine-tuning means taking a model that has already been trained, and training it further on a specific task. In this case, the researchers fine-tuned the models to optimise them to the task of producing a passage of prose based on a prompt describing that passage. They did this by taking lots of excerpts from books, using GPT-4o to summarise them, and using these excerpt-plus-summary pairs as training data during fine-tuning.

What they found is stunning. A model fine-tuned in this way on text from a handful of books will then regurgitate text from other books, which were not included in that fine-tuning set, to a far greater degree than would the original, non-fine-tuned model.

In other words, the process of fine-tuning the model to be able to expand summaries into full prose seems to unlock the ability to output copyrighted text that it has been trained on.

You can see some of the results below. Take, for example, the results for Never Let Me Go, which you can see on the top row of the figure. Using the standard, production version of GPT-4o, the longest block of text the researchers were able to show the model had memorised was 19 words. After fine tuning, this increased to 293 words - a 15x increase.

Or take another data point regarding the same book. Before fine-tuning, they found in the output zero blocks of text that copied more than 20 words from the book; after fine-tuning, they found 117 such blocks of text.

To really understand what is going on here, look at the example below. The text in the box at the top is a summary the researchers provided to the various models; the second box shows the output they got from the base GPT-4o model; the third box shows what the fine-tuned version of GPT-4o produced. Text in red is text that can be found, verbatim, in Betwen the World and Me by Ta-Nehisi Coates.

As you can see, the standard version of GPT-4o, when responding to this prompt, does not output text from the book - a casual user would assume it hasn’t memorised the passage in question. But the output from the fine-tuned model proves that it has.

Seeing this, it is clear: there is no argument that these models don’t memorise their training data.

Using this technique, the researchers were able to demonstrate that these models had memorised up to 85-90% of some books verbatim. And every model they tried exhibited significant memorisation of its training data. I recommend reading the full paper to truly grasp the extent of the memorisation occuring here - it is shocking.

This is incontrovertible evidence that large language models memorise some of their training data - and the amount they memorise seems to be far greater than was previously known.

Legal ramifications

AI companies have a tendency to claim that training does not involve memorisation. For instance, Google told the U.S. Copyright Office in 2023 that “there is no copy of the training data — whether text, images, or other formats — present in the model itself”. OpenAI said that “models do not store copies of the information that they learn from”, calling the belief that they do “a common and unfortunate misperception of the technology”.

The indisputable fact that models do store copies of their training data is of particular note because, as the paper says, copyright law is territorial. If a model is trained in the US on British authors’ works, for example - and if those authors can’t prove that any copying took place in the UK - then they will have difficulty bringing a lawsuit. It was this that led to the demise of the Getty v. Stability AI case in the UK.

But the fact that models retain copies of their training data changes the equation. If a model is made available in the UK, and that model retains copies of an author’s work, it is much easier to bring a claim in the UK. The AI company cannot say they are not making any copies of the work in the UK, since the model itself retains copies.

This finding, then, makes it much more likely that lawsuits against AI companies are brought outside the US. In the US, there is debate over the legality of training on copyrighted work without a licence, which will be decided in the courts. In other countries, such as the UK, there is no such debate. Commercial copying is forbidden except in specific circumstances. This is likely to be a major issue for AI comapnies.

Indeed, this is a major reason AI companies have been so keen to assure lawmakers that models do not retain copies - so that they cannot be sued in countries in which they make those models available. But, to be clear, they do store copies. This paper proves it.

Another significant legal ramification is on the fair use analysis in the US. The fourth factor in the fair use analysis is the effect on the potential market for or value of the work that is copied. It has been argued that training a model that cannot regurgitate its training data is more likely to be considered fair use, because the inability to regurgitate means the market for the original is less likely to be harmed. But, as this paper shows, the models can regurgitate. A model that outputs verbatim copies of what it is trained on is unlikely to score well on this factor.

A note on book piracy

The paper’s authors make one more interesting observation.

They took some of the copyrighted passages of text that they were able to extract from the models, and they searched common web-scraped datasets for these passages. Many of them weren’t there.

They then checked whether the books in question were present in the well-known pirate libraries LibGen or Books3. 80 of the 81 books they tested were indeed present in at least one of those pirate libraries.

What does this mean? The paper sums it up well:

This provides strong circumstantial evidence that the memorization observed in frontier models is unlikely to originate solely from content incidentally encountered through web crawling.

Expect this to be referenced in lawsuits - a lot.

What these researchers have demonstrated is hugely significant. AI developers are losing the argument on memorisation - as much as they like to portray their models as mere pattern-learners, the truth is that LLMs memorise. If they are trained on copyrighted text, they memorise copyrighted work.

This finding is a gift to writers and publishers suing AI companies. Expect to hear a lot about it in the months and years to come, as the legal battles escalate.