Memory Caching: Giving Recurrent Models Growing Memory

This episode examines "Memory Caching: RNNs with Growing Memory," a February 2026 paper proposing that recurrent models save checkpoints of their own memory as a sequence unfolds. The aim is to recover the recall that fixed-size states lose, at a cost between linear and quadratic. It sets up the problem with needle-in-a-haystack tests, then traces how linear attention, Titans and Deep Linear Attention treat memory as a small network trained during the forward pass. The paper's variants are introduced: summing cached memories, gating them per token, routing to a sparse few, and merging their parameters in "Memory Soup." The discussion ties Memory Soup to model soups and linear mode connectivity. Averaging weights only helps if the cached memories share a loss basin, so the episode treats that as the claim to test. Listeners get a clear explanation of why recurrent models forget and a skeptical way to judge whether this fix works.

Sources:

1. Memory Caching: Giving Recurrent Models Growing Memory

https://arxiv.org/pdf/2602.24281v1

2. Memory Caching: RNNs with Growing Memory — Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2026

https://scholar.google.com/scholar?q=Memory+Caching%3A+RNNs+with+Growing+Memory

3. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, et al., 2022

https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time

4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

5. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018

https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%29