A new test for AI progress: learn something and keep it

A model that answers correctly while the relevant facts remain in its prompt has demonstrated a useful ability. A model that retains new knowledge after that temporary support disappears has demonstrated a different one. Progress toward continual learning depends on measuring the distinction rather than treating every kind of memory as the same achievement.

Google Research’s October 27 sleep-framework blog discusses an engineered cycle of active processing and offline consolidation. The linked Language Models Need Sleep paper predates the blog: its arXiv history begins in June, with a July revision, and it notes an earlier OpenReview version. This is a recent explanation of an existing research line, not a newly submitted October paper.

The measurement question is decisive: after learning something new, what can the system still do, and what did retaining the knowledge cost? AGI Tracker has not reproduced the paper. The scorecard and examples below are proposed ways to answer that question.

Four mechanisms that can look like memory

An assistant may appear to remember because the information remains in its context. It may retrieve a stored document. It may consult a persistent note written by a previous session. Or its learned parameters may have changed. Each mechanism can be useful, but the evidence for one does not establish another.

The distinction matters when reading a retention claim. If the model receives the answer in a retrieved note, success demonstrates the retrieval-and-reasoning system under that condition. If the temporary context is removed and no external store is available, success supports a different claim. Neither should be promoted into an all-purpose statement that the model learns continuously.

A proposed evaluation should state exactly what persists between sessions: parameters, retrieval index, conversation summary, executable skill or some combination. It should also state what the evaluator resets. Without that record, two systems can appear to take the same test while carrying very different information into it.

Voyager is useful historical background because its agent uses a reusable skill library without fine-tuning the underlying language model. That is a concrete example of why system-level learning and parameter learning should be described separately.

The research contribution concerns consolidation

The sleep paper studies consolidation and self-generated practice within a specified learning framework. That is an algorithmic proposal, not evidence that ordinary deployed assistants automatically acquire durable new knowledge whenever they interact with a user. The implementation and training procedure matter.

The metaphor can help explain the motivation, but it should not carry the conclusion. Biological sleep is not an evaluation metric. The relevant questions concern what information moves between memory components, which parameters change and how those changes affect subsequent behavior.

A useful reading therefore replaces the metaphor with an experiment: expose the system to new information, perform the specified consolidation procedure, remove temporary support and measure what remains. Then test older capabilities under the same conditions. That sequence turns an attractive analogy into a falsifiable claim.

The older catastrophic-forgetting paper establishes the broader research problem of preserving earlier competence while learning new tasks. The important issue is not whether a method uses the word sleep. It is whether it improves the tradeoff between plasticity and stability under a credible test.

A retention score needs a timeline

A single before-and-after comparison can conceal how quickly learning decays. A system might retain a new fact immediately after consolidation and lose access to it after several later updates. Another might retain the fact but become less able to use it in a changed question.

A proposed study should evaluate at multiple points in the task stream. Record performance immediately after learning, after unrelated new material and after a deliberately interfering update. Keep the test items separate from the material used for rehearsal. Otherwise, the evaluation may reward repeated exposure to the same questions.

The Gradient Episodic Memory paper provides background on measuring transfer across a sequence of tasks. The broader principle is that the order of experience belongs in the experiment. A final score without a history can miss both forgetting and beneficial transfer.

For example, a hypothetical model learns a fictional company’s new policy, then learns an unrelated technical manual, then receives a revised version of the policy. The evaluator can ask whether it retains the first policy where still valid, adopts the revision where necessary and avoids mixing the two. That tests more than rote recall.

A proposed retention-versus-interference scorecard

The following dimensions are an editorial proposal, not a reproduction of the paper’s metrics. They are designed to make different forms of progress visible.

The table avoids a common shortcut: declaring success because the new-task score increased. A continual learner has to be assessed on the older tasks it was expected to preserve. A method that trades away essential old behavior may be unsuitable even when the average improves.

The appropriate weighting depends on the application. A personal assistant, a scientific analysis system and a model updated from public documents may have different retention requirements. Publishing the component measurements lets a reader make that judgment instead of inheriting an unexplained aggregate score.

Expansion changes the resource comparison

A method can preserve more knowledge by allocating more capacity. That can be a legitimate design improvement, but it changes the comparison. An expanding system and a fixed-capacity system do not face identical resource constraints.

For a proposed study, publish both a fixed-resource comparison and a practical expanding-system comparison where feasible. The first asks how efficiently a method uses a given budget. The second asks what a complete system can achieve when allowed the additional capacity its design requires.

The Chinchilla research is relevant background on why compute allocation and model size cannot be interpreted independently. For continual learning, the accounting also extends through time. A method’s early advantage may have a different cost after many consolidation cycles.

Record activated parameters, stored examples, external memory and cumulative update work. Do not count only the inference cost after learning has completed. The system had to pay to acquire and preserve the new capability, and that expenditure matters when comparing it with retrieval or periodic retraining.

A worked example: a changing technical manual

Imagine evaluating an assistant on a fictional machine manual. The initial manual specifies one operating procedure. A later update changes a component and introduces an exception. All names and facts are generated for the experiment so that pretraining knowledge cannot supply the answers. This is a proposed design.

First, test the model on the original manual using held-out questions that require applying the procedure. Then expose it to the update through the learning mechanism under study. Remove the temporary update context before testing retention, unless the claim specifically concerns a context-based system.

The next test should include three kinds of questions: unchanged rules, revised rules and situations where the exception applies. A model that simply memorizes the latest text may answer the changed rule correctly while incorrectly applying it everywhere. A useful learner must preserve the distinction.

Then introduce unrelated material and repeat the evaluation with new wording. This separates immediate recall from persistence and tests whether the learned information is accessible through a different query. Record whether the system asks for clarification when the version is unspecified.

Finally, compare with a retrieval-based assistant supplied with the same documents. If retrieval performs better under the available budget, that is an informative outcome. A parameter-learning mechanism does not become preferable merely because it seems closer to an intuitive picture of human memory.

Self-generated practice needs an external anchor

A system can generate additional examples to rehearse new knowledge. The measurement risk is that it practices its own mistakes or becomes better at the particular question forms it produces. Improvement on that self-generated distribution may not transfer to independent tests.

A proposed evaluation should therefore separate the generator of practice data from the author of the final test. It should retain the practice corpus and check for overlap with evaluation items. If an external teacher supplies privileged information, record what information the student receives directly and indirectly.

The original distillation paper supplies historical context for learning from another model’s behavior. The continual-learning question adds a time dimension: does repeated transfer preserve useful knowledge, and how does it interact with what the student already knows?

A useful ablation would remove or alter the practice stage while holding the rest of the procedure stable. Another would keep practice volume constant but change its diversity. These are proposed experiments intended to identify why a method works, not claims that the published paper omitted those analyses.

Retention is not always the desired outcome

A learner should not preserve every statement indiscriminately. Some information becomes obsolete, some is wrong and some is inappropriate to retain. An evaluation focused only on remembering more can reward behavior that makes the system less useful.

This is why correction and version handling belong beside retention. If a fictional policy changes, the model should adopt the revision under the correct scope rather than merely accumulate both versions. If evidence invalidates an earlier claim, persistence of the old claim is a failure, not a memory success.

A proposed test can distinguish forgetting from selective revision. Ask questions that explicitly specify the old version, the new version or the current rule. The desired answers differ. A model that can retrieve the historical rule while applying the current one has shown a more useful capability than a model that always repeats whichever statement it saw last.

These are system-design requirements rather than evidence of subjective understanding. They make continual learning operational: the goal is dependable adaptation to a changing information environment, with a traceable relationship between what was learned and what the system now does.

A missing answer can be a retrieval failure or a learning failure

Suppose the system gives the wrong answer after an update. The evaluator should not immediately conclude that the knowledge was erased. It may remain accessible under a different prompt, or the model may confuse which version applies. A useful diagnostic separates storage, retrieval and application, without treating a favorable rephrasing as permission to replace the original failed score.

For a proposed study, predeclare a small diagnostic set used only after the main evaluation. Ask a direct recall question, a version-specific question and a transfer question. Report the diagnostic outcomes separately. This can identify whether the method needs better consolidation or a better way to use what it retained.

The distinction is practically important. Improving durable storage may not repair a system that cannot choose between conflicting remembered rules. Improving retrieval may not help if the update was never incorporated. A definitive report should identify which part of the learning loop its evidence actually tests.

What would strengthen the progress claim?

The strongest follow-up would show sustained performance across a long, diverse stream with a realistic cumulative budget. It would include conflicting updates, held-out tasks and comparisons with simpler memory mechanisms. It would report where the method fails rather than only the most favorable retention result.

An independent reproduction would be particularly valuable because the learning procedure contains many choices. The model, update schedule, practice generation and memory allocation can all affect the outcome. Releasing enough artifacts to isolate those choices makes the work easier to build on.

Our existing AI-amnesia explainer introduces the underlying problem. Our guide to the three levers of AI progress explains why an algorithmic improvement still needs a resource denominator.

The sleep framework puts an important question back in view: can an AI system turn experience into durable, usable competence without erasing what it already needs? A definitive answer requires a timeline, a retention test and a full account of the cost. A single long-context demonstration cannot supply all three.

Sources

• Google Research sleep-framework blog, October 27, 2026
• Language Models Need Sleep, revised July 2026
• Overcoming catastrophic forgetting in neural networks
• Gradient Episodic Memory for Continual Learning
• Voyager
• Training Compute-Optimal Large Language Models
• Distilling the Knowledge in a Neural Network