Recurrent Latent Memory: Continual Learning Done Simplistic

中文 · Open the standalone illustration

Overview

Recurrent latent memory is a memory framework for continual-learning agents. Alongside exact working context, external persistent memory, and model parameters, it introduces a recurrent set of readable and writable continuous latent vectors. These vectors need not preserve history verbatim. They can carry fuzzy indices, abstract patterns, experience, and intuitions that guide later reasoning and precise retrieval.

During each memory cycle, the agent works with a system prompt, latent memory from the previous cycle, and the current exact working context. Once that context reaches a threshold, the framework inserts a set of <summarize> positions in one parallel prefill operation. Their final-layer hidden states directly become the next latent memory. The old state remains active until the new state is complete, while exact facts in external storage persist independently.

The framework is defined by a memory interface rather than one fixed network architecture:

$$ M_{t+1}=U_{\phi_t}(M_t,E_t;K_t) $$

Here, $M_t$ is the previous latent memory, $E_t$ is the evidence available for consolidation, $K_t$ is the latent-token budget for the next state, and $U_{\phi_t}$ is a trainable memory updater. The updater may be the same Backbone used for routine decoding, or a larger specialist model that runs less frequently.

This repository currently provides an interactive architectural illustration of the research proposal, not a trained continual-learning model. Capacities, tokens, and flows shown on the page are schematic configurations.

Why latent memory

For current agents, reading strings, writing strings, and search operations such as grep are atomic capabilities from which more complex behavior can be composed. This framework proposes that continual learning additionally requires latent-memory reads and writes as atomic capabilities.

People do not remember every experience verbatim. We often retain indices at different levels of abstraction, such as remembering where to find a recipe instead of memorizing the full recipe. Repeated experience also forms intuitions that may resist complete verbalization, such as an initial sense of which technique might solve a new mathematics problem.

Text notes are appropriate for facts that must be read exactly, audited, or shared, but not every compressed index or intuition has a natural textual representation. The atomic memory capabilities of a continual-learning agent can therefore include:

• Reading and writing exact strings in the current context and external records.
• Reading and writing continuous latent memory for fuzzy state that is difficult to express verbatim.
• Retrieving precise information from tools or persistent storage under the guidance of latent indices.

The corresponding training goals are to compress the minimum information that remains useful, update or forget latent state correctly, and make genuine use of that state in later tasks.

Four complementary forms of memory

Memory	Representation	Primary role	Lifetime
Exact working context	Text, image, audio, tool-result, and other tokens	Holds interactions that can be reread verbatim in the current cycle	Short-term, mutable, capacity-limited
Recurrent latent memory	$M \times d$ continuous values	Carries fuzzy indices, patterns, and experience across cycles	Long-term, updateable, capacity-controlled
External persistent memory	Exact records in a database or notes store	Preserves information that must be retrieved, verified, or traced exactly	Long-term, explicit, independent of context
Model parameters	Weights	Encode general capabilities and knowledge learned during training	Normally fixed within an inference-time memory loop

These forms of memory complement rather than replace one another. Latent memory represents where to look and what might work, persistent storage preserves exact content, working context carries the current explicit interaction, and model parameters provide general computation.

Illustrative architecture

The example places a flexible system prompt outside an illustrative 100K budget and divides that budget into:

• 10K latent-memory tokens from the previous cycle.
• Up to 80K exact working-context tokens.
• 10K reserved <summarize> positions.

The 10K value is only an example. More generally, the latent-state capacity is $M \times d$, the working context contains up to $C$ exact tokens, and $M$, $C$, and the update clock may be fixed, scheduled, or learned.

One complete memory cycle

1. Interact

The user message enters an empty working context as one parallel prefill chunk. The agent then decodes a tool call token by token, and the tool result returns as another complete prefill chunk. If a fact must be retained exactly, the agent decodes a storage call that writes the content and its source into persistent memory before decoding the final reply.

2. Reach the threshold

When latent memory and exact working context reach the consolidation threshold, the framework fills all <summarize> positions simultaneously. In the example, 10K latent tokens and 80K working-context tokens form the 90K pre-consolidation input before 10K consolidation positions are added.

3. Consolidate

The Backbone processes the complete sequence as one parallel prefill. Final-layer hidden states at the <summarize> positions directly form the staged $M_{t+1}$, with no depicted projection and no persistent per-layer KV cache. The old $M_t$ remains valid until this operation finishes, avoiding an in-place circular overwrite.

4. Turn over

Once the new latent state is completely staged, it replaces the old state through a versioned, atomic handoff. Exact working context and <summarize> positions clear, the system prompt remains fixed, external persistent memory survives independently, and the agent enters the next cycle.

Prefill, decoding, and tool boundaries

• User messages, tool results, storage acknowledgments, and <summarize> positions are prefilled as complete chunks.
• Tool calls, storage calls, and natural-language replies are decoded autoregressively, one token at a time.
• A completed tool call leaves the Agent Output Window and enters the tool endpoint; returned data enters working context directly and does not transform old output tokens into another type.
• Exact content that must survive is written to persistent storage instead of relying only on latent state.
• Visible cells are aggregate representations. They neither correspond to a specific tokenizer nor claim that one visible cell equals one actual model token.

Training the mechanism

One direct starting point is RNN-style chunked pretraining: process one context chunk, produce the next latent state, and recursively use it with later chunks. This provides dense sequential training signals for compression, retention, forgetting, and retrieval, at the cost of substantial recursive compute.

Current agent-training tools can then optimize the memory policy further through reinforcement learning, outcome supervision, correction signals, teacher distillation, and cross-cycle retrieval tasks. Training should reward not only what the model remembers, but also correct forgetting, resistance to latent drift, citation of exact external evidence, and selection of valuable information under a limited budget.

Relationship to other architectures

Architecture	Mutable state	Information directly available to one update	Key distinction
Conventional RNN	Typically a $d$-dimensional hidden state	Current token and previous hidden state	Updates once per token with a narrow window of exact evidence per update
Recurrent textual/discrete memory	$M$ vocabulary tokens or discrete codes	Previous discrete state and current context	Raw channel of at most roughly $M\log_2
Recurrent latent memory	$M \times d$ continuous values	Previous latent state and up to $C$ exact context tokens	Consolidation can be better informed and may yield higher-quality compression, although quality still depends on the training objective
Inference-time weight adaptation	Potentially $	\theta	$ model parameters

Recurrent latent memory does not attempt to replace inference-time weight adaptation. If reliable online weight learning becomes available, latent memory can serve as a frequent fast-learning loop while weight updates provide an infrequent slow-learning loop.

Extensible memory policies

The shared update interface can learn varied memory behavior without requiring an LSTM-style gating architecture:

• Learned retention and forgetting: The updater can learn to copy, weaken, overwrite, or erase latent signals. Explicit gates may serve as an inductive bias, but they are not required by the framework.
• Multiple update rates: Working context provides instant memory, latent state updates by task or threshold, and large-scale consolidation can run hourly or daily.
• Heterogeneous readers and writers: A small model can handle bandwidth-bound token decoding, while a larger model performs higher-quality updates during the more parallel prefill stage.
• Outcome-aware reflection: An infrequent updater can read archived cycles, tool outcomes, corrections, and task success or failure to perform deeper consolidation resembling reflection or dreaming.
• Adaptive memory size: The model can be pretrained across multiple values of $K$, or a controller and reinforcement learning can select how many latent tokens the next cycle needs.

Heterogeneous models must share a stable latent interface or learn an explicit bridge. Very large latent states also increase attention and storage costs, so hierarchical, sparse, or retrieval-based access may be necessary instead of assuming dense attention over every slot.

Open research questions

• Which objectives preserve information that is genuinely useful for future tasks instead of optimizing only short-term reconstruction?
• How should quantization error, latent drift, catastrophic forgetting, and false memories be measured over hundreds or thousands of updates?
• What information should enter latent memory, exact persistent storage, or model weights?
• How can reader and writer models with different sizes or architectures share, migrate, and version latent state?
• Do adaptive $K_t$, learned update clocks, and deep consolidation provide enough benefit to justify their additional training and inference compute?

Repository and usage

illustration/ contains the bilingual interactive illustration, state-machine tests, and detailed development documentation. illustration/dist/index.html is the complete standalone file generated with Vite and vite-plugin-singlefile; it can be copied or opened directly in a browser.

cd illustration
bun install
bun dev
bun test
bun run lint
bun run build
bun run build:single

bun run build runs the standard Next.js production build and creates .next; this is also the default Vercel build path. bun run build:single uses Vite to generate and verify the standalone dist/index.html. The page simulates tools, storage, and memory flow; it makes no external requests and writes no real data.

For Vercel, set the project Root Directory to illustration/ and use the Next.js framework defaults without Build Command or Output Directory overrides. Vercel will run the standard build script and deploy its .next output.

Citation

If you use this project in research or other published work, use GitHub’s Cite this repository control or the metadata in CITATION.cff. GitHub can generate APA and BibTeX citations from that file.

License

This project is licensed under the GNU Affero General Public License v3.0.