Imagine asking a model to forget a collection of documents. You run an unlearning procedure, test it, and find that it can barely answer questions about them anymore. Success?

Then you briefly fine-tune the model, and much of its old performance returns.

That is the problem Xiaoyu Xu and colleagues investigate in Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. Their finding is uncomfortable: a model can appear to forget while retaining internal structures that make recovery surprisingly easy. A bad answer after unlearning does not, by itself, establish that the information is gone. [1, Section 3]

Machine unlearning aims to remove the influence of selected training data. The ideal reference is a model trained without that data in the first place. Because retraining an LLM from scratch is expensive, researchers usually modify the existing model and evaluate whether it has stopped reproducing the targeted knowledge while preserving other capabilities.

The authors call the targeted data the forget set and the remaining data the retain set. Both matter. Removing a capability by breaking the entire model would be a poor substitute for selective forgetting.

The difficulty is that we mostly observe outputs. When a model fails a question, we cannot immediately tell whether it has lost the relevant knowledge, become less willing to produce it, or suffered a broader disruption in how it turns internal features into answers.

Think of a library whose catalogue has been scrambled. Finding a book becomes difficult, but the shelves may still hold it. This is only an analogy; neural networks do not store documents as books. It captures the distinction the paper tests: loss of access can resemble loss of information.

To investigate, the authors follow models through three stages: original, unlearned, and relearned. Their primary experiments use Yi-6B and Qwen2.5-7B, with scientific abstracts, code, and mathematical reasoning data. They test several unlearning methods, including gradient ascent, which pushes the model toward higher loss on targeted examples, and methods that also try to protect retained capabilities. They examine both one removal request and sequences of requests. [1, Section 3.1]

Relearning is the crucial stress test. The researchers briefly fine-tune the modified model with a limited amount of data, rather than rebuilding it from its full pretraining corpus. Recovery data can come from the forget set itself, retained data, or unrelated data.

Using the forget set is an intentionally strong recovery scenario. It also introduces an important caveat: teaching a model the targeted examples again could create new learning. Recovery alone therefore cannot prove exactly what survived. The paper strengthens its investigation by examining the model's internal representations alongside its behavior.

One result makes the distinction vivid. In a continual-unlearning experiment with Yi-6B and gradient ascent, forget-set accuracy fell from an original 78.9% to approximately 9.1%. After relearning, it reached approximately 80.0%. Retain-set accuracy also recovered from 6.2% to 65.0%, close to the original 65.5%. The apparent collapse was largely reversible. [1, Tables 3 and 4]

So how do you look beneath the answers?

The authors collect activations, the internal numerical representations produced as inputs pass through the model's layers. They compare these before unlearning, afterward, and after recovery using complementary tools:

  • PCA similarity and shift track whether dominant patterns remain aligned and how far representations move.
  • Centered kernel alignment, or CKA, compares the structure of activation patterns between model states.
  • Fisher information estimates how sensitive the model's loss is to changes in individual parameters.

They also summarize representational movement with a mean PCA distance. These measurements are diagnostic clues, not a detector that declares a particular fact deleted. [1, Section 4]

In recoverable cases, the internal representations often remain relatively close to their original structure or return toward it after relearning. More aggressive updates can produce larger, persistent changes and poorer recovery.

That creates two separate questions: Can the targeted behavior return? And how much unrelated capability was damaged?

  • Reversible, little collateral damage: targeted performance drops but is readily restored.
  • Reversible, broad damage: the model deteriorates widely, then recovers.
  • Irreversible, broad damage: recovery fails, but useful capabilities are also lost.
  • Irreversible, little collateral damage: the desired outcome, durable and selective forgetting.

The last combination is particularly difficult. Increasing the strength or frequency of unlearning can make recovery harder, but it can also damage retained capabilities. A model that no longer knows much of anything has not demonstrated precise deletion. [1, Sections 4 and 5]

The recovery data matters too. In the authors' sample-efficiency experiments, fine-tuning on forgotten examples restored performance most effectively. Retained or unrelated data produced slower, weaker recovery. It would therefore overstate the results to say that any harmless fine-tuning always brings everything back. [1, Appendix A.4.2]

The paper does identify a case of seemingly irreversible, relatively selective forgetting using a method that focuses updates on influential parameters. But recovery there is restricted to particular data sources and budgets, and retained performance still declines. “Irreversible” means recovery failed under the tested protocol, not that every possible future recovery attempt has been ruled out. The authors explicitly acknowledge that their experiments provide no formal privacy guarantee. [1, Section 5.4; Appendix A.1]

For engineers, the practical lesson is to evaluate unlearning over time. Test targeted behavior, check retained capabilities, attempt recovery under stated constraints, and inspect internal changes where feasible.

A model's silence is useful evidence about its current behavior. Calling that silence deletion requires a much stronger case.

Reference

[1] Xiaoyu Xu, Xiang Yue, Yang Liu, Qingqing Ye, Huadi Zheng, Peizhao Hu, Minxin Du, and Haibo Hu. Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. arXiv:2505.16831, version 3, May 16, 2026. This explanation follows the supplied PDF. Paper and version history · Authors' analysis code.

Topics: #MachineUnlearning #LLMs #AIResearch #Relearning #ModelEvaluation #AIPrivacy