Weights That Keep Learning

Transcript

Cold open: the network that forgot how to add 0:07

Maya Picture a small neural network in nineteen eighty-nine. It's been taught simple addition. Specifically, all the "ones" facts. One plus one, two plus one, six plus one, and so on. It gets every one of them right.

Theo A very modest genius.

Maya Very modest. Then the researchers teach it the "twos" facts. Two plus two, two plus three, and so on. They train on just the new facts, which is how most learning in the world actually arrives: one thing after another. And after a single pass over the twos, they go back and test the old ones facts.

Theo And it's forgotten some.

Maya It's gone from one hundred percent correct to fifty-seven percent. After a second pass, thirty percent. And the mistakes aren't random. Ask it six plus one, and the answer it produces looks like eight. Ask it five plus one, and you get seven. It has started answering every old question as if it were a twos question.

Theo So the new lesson didn't sit next to the old one. It wrote over it.

Maya That's the finding. Michael McCloskey and Neal Cohen, "Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem." And they had a lovely, slightly chilling phrase for it. They said the forgetting looked "more like retrograde amnesia than retroactive interference."

Theo Meaning it wasn't the ordinary fuzziness people show when a new list of words muddles an old one.

Maya Right. People do forget when they learn new things, but gradually. This was more like a wipe. And here's the line I want to plant early, because it's the thread of the whole episode. The authors were careful. They said interference was catastrophic in some specific networks. They explicitly did not claim every network must forget like this.

Theo Which, thirty-odd years later, is still the honest state of the field.

Maya Welcome to The Reward Signal. I'm Maya.

Theo And I'm Theo. Last time we talked about what happens when you optimise a model for how much people like it. Today is a different itch. The language models most of us use don't change their weights when you talk to them. They can be retrained later, but a conversation changes the context, not the model. Today's question is what it would take for the weights themselves to keep learning, safely, for the rest of the model's life.

Maya Continual learning. Or lifelong learning, or online learning, depending on which decade you got your PhD in.

Theo And I'd argue it's two problems wearing one coat. The first is forgetting: learn something new without destroying something old. The second gets far less attention: keep the ability to learn at all. It turns out networks can quietly lose that too.

Maya So we'll ask three questions, in order. Can a network keep what it knew? Can it still learn something new? And when it does learn, does that knowledge outlast the conversation?

Theo Twenty papers, nineteen eighty-nine to the end of twenty twenty-five. And for each one, we'll try to say exactly what was shown, on what, and what wasn't.

Maya Because this field has a habit of headline numbers that come with an asterisk the size of a house.

Theo Let's go back to the little adding machine first, because McCloskey and Cohen tried to fix it, and how they failed tells you a lot.

Defining the problem, and three ways to measure it wrongly 3:49

Maya The arithmetic network wasn't their only experiment. They also simulated a classic memory study from human psychology. You learn a list of nonsense syllables, each paired with a word. Then you learn the same nonsense syllables paired with different words.

Theo The setup psychologists call A-B, then A-C.

Maya Exactly. People in the original human study, by Barnes and Underwood, still recalled about fifty-two percent of the old pairings after twenty trials on the new list. Interference, but partial. The network, after just three trials on the new list, went from one hundred percent on the old list to zero.

Theo Zero. And it had only learned about twenty percent of the new list by then.

Maya So it had thrown away the old knowledge before it had even acquired the new. Now, the authors are honest that this is a rough simulation. Scoring and training-to-criterion differ between the human study and the network, so it isn't a clean human-versus-machine race. But the gap in shape is the point.

Theo And then they tried to fix it.

Maya They tried a lot. Changing the number of hidden units. Different target activations. Massively overtraining the first list. Local codes instead of distributed ones. None of it removed the effect in the networks they tested. A lower learning rate kept some old answers, but only under a very generous scoring rule, and at a huge cost in training time. And the one I find most instructive: they tried freezing the weights that had learned the old list.

Theo Which sounds like it should simply work. Lock the old knowledge, learn the new on fresh weights.

Maya It didn't, because the new weights still fed into the same output units. Every pathway ends at the same answer neurons, so changing any of them changes the old answers. Freezing alone isn't isolation. That idea comes back later in a big way.

Theo Their diagnosis was geometric, and it's the cleanest mental model for the whole field. For every input and answer pair, there's a region of weight settings that gets it right. Train on two examples together and the optimiser heads for where those regions overlap. Train on the second one alone and nothing stops it walking straight out of the first region.

Maya In their words, "There is nothing to prevent" it. That's the problem in one sentence. Gradient descent only sees the data in front of it.

Theo Which is why the most boring fix in the world works. If you shuffle all the data together and train on everything at once, forgetting mostly doesn't happen. The trouble is that real data arrives in order and the old data isn't always around.

Maya Okay, jump forward thirty years. By two thousand nineteen there were dozens of methods, all reporting results on the same toy benchmarks, especially handwritten digits split into tasks. And Gido van de Ven and Andreas Tolias wrote a short paper called "Three scenarios for continual learning" that I think should be required reading before anyone reads any other paper in this episode.

Theo Because the same benchmark can be three completely different exams.

Maya Right. Take split MNIST. You learn zero versus one, then two versus three, and so on, five tasks. Now at test time, you're shown a digit. Scenario one, task-incremental: the examiner tells you which task it's from, so you only have to say "the first or the second of this pair". Scenario two, domain-incremental: you're not told the task, but you only need to say "first or second of whichever pair". Scenario three, class-incremental: you're not told anything, and you must say which digit, zero to nine.

Theo Same training, same images, three different questions.

Maya And the results are startling. Elastic weight consolidation, which we'll meet properly in a minute, scored ninety-eight point six percent in the first scenario. In the third, twenty point zero one percent.

Theo On ten digits, guessing gets you ten percent. So that's twice chance, and nowhere near useful.

Maya Nowhere near. Learning without Forgetting shows the same collapse. Meanwhile the methods that replay old digits, either generated ones or a store of real ones, stay above ninety percent in that hardest scenario.

Theo So "this method fails" is scenario-specific. EWC does beautifully when you tell it which task it's on.

Maya And the authors are careful about their own result. Generating fake MNIST digits is unusually easy, so it's an open question whether replay stays that strong on harder data. They also admit they chose hyperparameters using the tasks' test sets, which they flag openly. The lasting contribution isn't a winner. It's the question every result in this episode should answer: what was the model told at test time?

Theo Write that on a sticky note. We'll keep pulling it out.

Protecting the weights: LwF, progressive nets, EWC, synaptic intelligence 9:18

Theo Let's do the first family of fixes: protect what the network already knows. And I'll start with a paper whose name is a promise it doesn't quite keep. Zhizhong Li and Derek Hoiem, "Learning without Forgetting", European Conference on Computer Vision, twenty sixteen.

Maya Great name. Bold name.

Theo The motivating scenario is very practical. You've got an image recogniser on a building site that already checks hard hats and safety vests. The site manager wants it to also spot bad footwear. But the original training images are proprietary, or lost, or just enormous. You can't retrain on everything.

Maya So you only have pictures for the new task.

Theo Only new pictures. And the trick is lovely. Before you change anything, you run the new task's images through the old network and record what it says about them, in its old categories. Then, while you train the new skill, you add a penalty for drifting away from those recorded answers. The old network becomes its own teacher, on images it was never trained on.

Maya Distillation, but onto yourself.

Theo And it often works. Going from ImageNet objects to classifying bird species, it kept fifty-four point seven percent on the old task, against about fifty-one for plain fine-tuning, and did slightly better on the new task as well. The weakness is in the premise. You're preserving the old network's answers on new pictures, as a proxy for its answers on old pictures. If the two kinds of image look nothing alike, the proxy breaks. A scene recogniser learning bird species lost three point eight points on its old task. Training on both together lost only one and a half.

Maya So learning with somewhat less forgetting.

Theo Less forgetting than the alternatives they tested, which is a real achievement. Not none.

Maya The opposite philosophy came out the same year from Andrei Rusu and colleagues: "Progressive Neural Networks". Their attitude is, why risk it? When a new task arrives, freeze the old network completely and add a whole new column of layers beside it. The new column can read the old column's features through side connections, but nothing flows backwards.

Theo That's the isolation McCloskey and Cohen's freezing didn't have.

Maya Exactly. There, old and new weights shared the same outputs. Here each task has its own column and its own output. Old skills are preserved by construction, as long as you pick the right column. They tested it on reinforcement learning: altered versions of Pong, a set of Atari games, and three-D mazes where you collect apples and strawberries and avoid mushrooms and lemons.

Theo The fruit-based peril.

Maya And the transfer was real but mixed. On Atari it helped on eight of twelve target games, with two cases of negative transfer, where fine-tuning the whole network helped on only five. But read the costs. The paper itself says parameters grow quadratically with the number of columns. And at test time, you need to be told which task you're on, to choose a column.

Theo So it keeps the old skill perfectly, but someone still has to tell it which skill to use. Sticky note.

Maya Right. And results were reported by averaging the top three of twenty-five runs, which the authors state openly, but which flatters any method.

Theo Now the paper everyone cites. James Kirkpatrick and colleagues, "Overcoming catastrophic forgetting in neural networks", twenty seventeen. Elastic weight consolidation, EWC.

Maya And it comes with its own metaphor.

Theo It does, and it's a good one. After task A, save the weights, and estimate how much each weight matters to task A. Then while you learn task B, attach every weight to its old value with a spring. The authors write that "the stiffness of this spring should not be the same for all parameters." Important weights get stiff springs. Unimportant ones are free to move.

Maya And importance comes from where?

Theo From the Fisher information, roughly how sharply the old task's performance depends on that weight. Mathematically it's a quadratic penalty with a per-weight strength. On shuffled-pixel digit tasks, it held on to earlier tasks much better than ordinary training, or a penalty that treats every weight the same.

Maya But the Atari result needs the asterisk.

Theo The Atari agent wasn't just EWC. It also had game-specific parameters, a learned module to recognise which game it was playing, and separate replay buffers for each inferred game. And it still didn't reach the scores of ten separately trained agents. So when people say EWC learned many Atari games without replay, the paper doesn't support the "without replay" part.

Maya There's another detail I love. They tested whether moving weights in the directions the Fisher said were unimportant would be harmless. And the paper says, essentially, "Empirically, however," it wasn't. The importance estimate is a local approximation, and the authors call that a significant weakness themselves.

Theo Then a few months later, Friedemann Zenke, Ben Poole and Surya Ganguli: "Continual Learning Through Synaptic Intelligence." Same idea, springs on important weights, but a smarter way to measure importance.

Maya Instead of computing it at the end of a task, they track it during training.

Theo Right. At every step, you know how much each weight moved, and how much that movement reduced the loss. Add that up along the whole journey and you get a credit score for each weight: how much of the learning it did. Normalise by how far it ended up moving, and you have an importance that's computed online, locally, per weight.

Maya Hence "synaptic intelligence". And they're careful there too. They write that they make no claim biological synapses behave like the intelligent synapses of their model.

Theo It works on the benchmarks they test, with one caveat: on the split tasks, each task had its own output head, so the model only ever chose between that task's answers. And one small but telling detail. In their main comparison plot, the EWC curve is "extracted and replotted" from the EWC paper, not rerun under identical conditions.

Maya Which tells you how much of this field runs on comparing against each other's figures.

Replay: iCaRL, GEM, tiny memories, dark experience 16:17

Maya The second family is less elegant and, spoiler, often stronger. Keep some old experience around, and rehearse it. Replay.

Theo Which is what McCloskey and Cohen's geometry predicts. If the problem is that the optimiser can't see the old data, show it some old data.

Maya The landmark here is iCaRL, from Sylvestre-Alvise Rebuffi and colleagues, also twenty seventeen. Incremental classifier and representation learning. And it opens with a child at the zoo, learning new animals without forgetting the pet it has at home.

Theo That's the hard scenario, isn't it? Class-incremental. No one tells the child "this question is about zoo animals".

Maya The hardest of van de Ven's three. iCaRL keeps a small, fixed budget of real old images, chosen so that each class's stored images average out to roughly the class's typical features. When new classes arrive, it trains on the new images plus the stored ones, while also distilling the old network's outputs. And then it classifies in an unusual way: not with the network's output layer, but by asking which class's stored examples, averaged, look most like this new image in the current feature space.

Theo Nearest prototype.

Maya Nearest prototype. On a hundred CIFAR classes, arriving ten at a time, it averaged sixty-four point one percent, against forty-four point four for a version of Learning without Forgetting with no stored images. The authors are frank that the key ingredient is the exemplar images. Their ablation even found that when classes arrive two at a time, distillation can slightly hurt.

Theo And iCaRL still trains for seventy epochs on each new batch of classes. "Fixed memory" is not "cheap".

Maya Good point. Then Gradient Episodic Memory, GEM, from David Lopez-Paz and Marc'Aurelio Ranzato, also twenty seventeen. It uses its stored examples in a cleverer way. It doesn't train on them directly. Every time it computes an update for the new task, it checks: would this step increase the loss on any old task's memory? If yes, it projects the update to the nearest direction that doesn't.

Theo So old tasks get a veto over every step.

Maya A veto, solved as a small optimisation problem sized by the number of tasks, not by the millions of weights. And a nice side effect: the update is allowed to improve old tasks, not just preserve them. They measured backward transfer, the change in old-task accuracy caused by later learning, and on CIFAR it was slightly positive.

Theo And does the veto actually pay off?

Maya On twenty shuffled-pixel digit tasks, seen once each, it finished at eighty-two point six percent, against sixty point two for plain training. On CIFAR, though, each task had its own classifier head, so it only had to choose among the classes in the task it was given, not every class it had ever seen.

Theo And then my favourite type of paper. The "wait, what if the simple thing works" paper. Arslan Chaudhry and colleagues, twenty nineteen: "On Tiny Episodic Memories in Continual Learning".

Maya Because the GEM paper had a line in it.

Theo It did. GEM dismissed training directly on the stored examples, saying it "results in overfitting". Chaudhry's team quote that line and test it. They take each new mini-batch, add a mini-batch sampled from a small memory of old examples, and take an ordinary training step. That's it. Experience replay.

Maya How small is tiny?

Theo On split CIFAR, one example per class. Eighty-five memory slots in total. And with that, replay scored fifty-six point two percent average accuracy against forty point six for plain fine-tuning, and slightly above A-GEM, the faster version of GEM. And it was cheaper to run than A-GEM, too.

Maya So the overfitting worry was wrong?

Theo Half wrong, and the half that's right is interesting. When they trained only on the stored old examples, accuracy on the old task fell from eighty-five percent to forty. It did overfit. But mixing memory with new-task data mostly fixed that; the fresh data acts like a regulariser. Mostly. They built an almost adversarial pair of tasks, rotated digits, where an upside-down two looks like a five, and there it could still hurt.

Maya One more replay paper, because it adds a genuinely good idea. Pietro Buzzega and colleagues, NeurIPS twenty twenty: "Dark Experience for General Continual Learning: a Strong, Simple Baseline". Dark experience replay.

Theo Dark as in "dark knowledge", the phrase from Geoffrey Hinton and colleagues that the paper cites, for the information hidden in a model's soft scores.

Maya Exactly. Instead of storing an old image and its label, you store the image and the network's raw scores for it, its logits, at the moment it was seen. Later, you replay the image and ask today's network to produce the same scores. The label tells you the right answer. The scores tell you how the old network saw it: what it thought it looked a bit like.

Theo The answer key and the old marked exam paper.

Maya That's our analogy, not theirs, but yes. The plus-plus version stores both. With a buffer of five hundred images on split CIFAR-10, in the hardest class-incremental setting, it scored seventy-two point seven percent against fifty-seven point seven for ordinary replay. But two caveats. When the task was given at test time, ordinary replay and dark replay were essentially tied, around ninety-three and a half. And those CIFAR runs trained for fifty epochs per task, not one pass.

Theo Which is the theme of this section. Replay is boring, it needs storage, and it keeps beating the elegant methods. Storing a few examples per class matters more than most of the clever regularisers.

Maya Hold on to that, because the large language model people arrive at the same answer, at a vastly larger scale.

The other half of the problem: loss of plasticity 23:06

Theo Everything so far has been about the first problem: don't forget old things. Now the second problem, and I think it's the most underrated paper on our list. Shibhansh Dohare, Rupam Mahmood, Richard Sutton and colleagues, in Nature, August twenty twenty-four. "Loss of plasticity in deep continual learning."

Maya Sutton as in the reinforcement learning textbook.

Theo That Sutton. And their question isn't "does the network remember task one?" It's "after a thousand tasks, can the network still learn task one thousand and one as well as a fresh network could?"

Maya Which you'd assume, surely. Training more shouldn't make you worse at training.

Theo That's what everyone assumed, and it's the experiment nobody ran at length, because it's expensive. They note a systematic study with large language models "would not be possible today", because a single run costs millions of dollars. So they built long-running tests from classic benchmarks instead. The main one they call Continual ImageNet. Every task is a binary choice between two ImageNet classes. Their example: the first task might be cats versus houses, the second stop signs versus school buses.

Maya Thousands of little yes-no tasks.

Theo Five thousand of them, on small thirty-two by thirty-two images, six hundred training images per class, and the network gets two hundred and fifty passes over each task's data. Plenty of time. Early on, a well-tuned network reaches up to eighty-eight percent on the test set. But by around the two thousandth task, for every step size they tried, it has lost a lot of its ability to learn, ending near or below a linear network.

Maya A linear model. No hidden layers at all.

Theo And that's how they define their big claim, which is deliberately provocative. Their discussion says that in settings where learning must continue, "deep learning does not work". And they define the terms carefully. By deep learning, they mean the existing standard algorithms. By not working, they mean that over time "they fail to learn appreciably better than shallow networks."

Maya That's a strong sentence to put in Nature.

Theo It is, and it's scoped. Not every model, not every fine-tune. Standard methods, run long enough. And they found it in a second, very different setting: CIFAR-100, a residual network, classes added five at a time. Crucially, this network keeps training on the old classes too, so forgetting isn't the issue. It's compared with a network trained from scratch on the same classes. After forty classes the continually trained one starts falling behind, and by the end it's about five percent worse.

Maya Same data, worse result, just because it had been training longer.

Theo And then reinforcement learning. A simulated four-legged robot, the Ant, with the ground's friction changing every two million steps. Standard PPO relearns after each change, which gives the reward curve a sawtooth shape, but the teeth get smaller. And even with friction held constant for fifty million steps, average performance rose for about three million steps then collapsed. After twenty million, the ant was "failing every episode".

Maya Without anything changing.

Theo Nothing in the world changed. The network just lost the ability to learn. They look inside and find three things that keep co-occurring with it: units that have gone dead and never fire, weights that have grown large, and the representations collapsing in rank, using fewer effective dimensions. They're careful to say these are correlates rather than a proven single cause.

Maya And the surprising bit, for practitioners.

Theo Some of the standard tools made it worse. Adam, dropout and normalisation "actually increased loss of plasticity" in their tests. Whereas plain L2 regularisation, pulling weights gently towards zero, often helped.

Maya And their own fix is my favourite idea in the paper, because it's so simple. They call it continual backpropagation. Ordinary training, plus a slow, constant renovation. Each hidden unit gets a running usefulness score: roughly how active it is, times how much its outgoing weights matter. Every so often, the least useful mature unit in a layer is retired and replaced with a fresh one. Its outgoing weights start at zero, so the swap doesn't disturb what the network currently predicts, and new units are protected for a while so they can mature.

Theo And "every so often" really means rarely. The paper says it can work out to "only one unit is replaced after hundreds of updates".

Maya So it's not a restart. It's replacing a few bricks at a time, forever.

Theo And it works in their tests. Continual backpropagation, L2, and a related method called Shrink and Perturb all held performance across the five thousand ImageNet tasks. On CIFAR, continual backpropagation closed the gap with retraining from scratch, finishing at seventy-six percent.

Maya All right. Where's the asterisk? There's always an asterisk.

Theo Several, mostly from the authors. The utility score is a heuristic, and they say so. The ImageNet tasks reset the output layer at each switch, which quietly tells the network when a task changes. Not quite our sticky note, since it's training-time help rather than a label at test time, but it's help all the same. In the reinforcement learning experiments, continual backpropagation was combined with L2 and a tuned Adam, so not all the gain belongs to unit replacement. And none of it was tested on large language models.

Maya But the conceptual point survives the caveats. A system that learns forever needs two properties: it mustn't forget, and it mustn't fossilise. Most of the field only measured the first.

Theo And fixes for one can hurt the other. EWC's stiff springs are a way of reducing plasticity on purpose.

Fast weights: from 1992 to linear Transformers 29:43

Maya Now I want to switch traditions, because "weights that keep changing" has a second meaning. And there's a little thread back to nineteen eighty-nine here: McCloskey and Cohen's own reference list includes a nineteen eighty-seven paper by Geoffrey Hinton and David Plaut called "Using fast weights to deblur old memories". The fast-weights idea is about weights that change not over months of new tasks, but while the model is reading, within a single sequence.

Theo Weights as working memory.

Maya And it starts with Jürgen Schmidhuber, nineteen ninety-two, in Neural Computation: "Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks". At the time, if you wanted a network to remember something from earlier in a sequence, you used a recurrent network, which keeps a running pattern of activity.

Theo The memory lives in the neurons' activations.

Maya Schmidhuber's proposal: put it in the connections instead. Two networks. A slow one, trained normally by gradient descent, watches the input stream and emits instructions for changing the weights of a second, fast network. The fast network's weights can change instantly, and they stay changed until told otherwise. So they hold information over time. In his words, a single weight, instead of a full-fledged unit, may be enough to store temporal information.

Theo And the experiments are charmingly small.

Maya Very. The first is a flip-flop task: random events A, B and C stream past, and the network must switch on its output the first time a B appears after an A. The solution it learns: when an A arrives, the slow network turns up the weight on the fast network's B input. When a B comes, it fires and the weight is turned back down. The waiting state is literally a weight.

Theo And the second one has a parking lot.

Maya A toy parking lot. Inputs announce which of three slots your car is in, mixed with distractors, and then an input literally called "where is my car" asks the question. The network solves it by creating a temporary strong connection from the question input to the right slot output. Solved within six thousand time steps. To be clear, these are binary symbols, not pictures of cars, and there's no benchmark table or scaling study. It's a proof of concept.

Theo And one thing to underline: this is memory across time steps in a sequence. It does nothing for the McCloskey problem of keeping old tasks after training on new ones.

Maya Agreed, different problem. But it set up a story that came back almost thirty years later. Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, twenty twenty-one: "Linear Transformers Are Secretly Fast Weight Programmers".

Theo A title with an attitude.

Maya It's a real result, though. Standard attention keeps every previous token's key and value around, so its cost grows with the sequence. Linear attention, a family of cheaper variants, instead compresses everything into one fixed-size matrix. Each new token adds the outer product of its value and key into that matrix, and a query reads from it. The paper shows that this is exactly Schmidhuber's slow network writing to fast weights. The matrix is the fast weight.

Theo Linear attention specifically, not ordinary softmax attention.

Maya Right, and the authors are precise about that. Then they ask what goes wrong with this memory, and the answer is capacity. If you only ever add associations, they start interfering. In a synthetic retrieval test with sixty-four-dimensional keys, ordinary linear attention began making errors at around sixty stored associations.

Theo So even a fixed-size memory made of weights runs out of room.

Maya In that setup, quickly. Their fix comes straight from old learning theory, the delta rule. Before writing a new value for a key, ask the memory what it currently returns for that key. Then write only the difference, scaled by a learned write strength. So instead of piling a new association on top, you correct the old one.

Theo Error-correcting memory, rather than accumulating memory.

Maya And it helps where you'd expect. In a toy task where keys get reassigned new values and you must return the most recent one, the simple additive rule fails, and the delta rule does well. In real language modelling on WikiText, it brought a ninety-million-parameter model's perplexity down from thirty-three to thirty-one and a half against additive linear attention. Though a regular Transformer of that size was still better, at twenty-nine point six.

Theo So the delta rule fixes linear attention's memory, but doesn't beat full attention.

Maya Not in that comparison. And with a much bigger state, Transformer-XL still won on perplexity. Where this matters for our story is the next step. If a matrix of fast weights updated by a delta rule is a memory, why stop at a matrix? Why not make the fast weights a whole small neural network, and update it with real gradient descent while the model reads?

Theo Which is where test-time training comes in. And it has its own origin story, in computer vision.

Learning at test time: TTT, TTT layers, Titans 35:28

Theo Yu Sun and colleagues, twenty twenty, at the International Conference on Machine Learning: "Test-Time Training with Self-Supervision for Generalization under Distribution Shifts." Their opening line is the idea: "We turn a single unlabeled test sample into a self-supervised learning problem."

Maya Meaning the model trains on the very image it's about to classify.

Theo Without its label, which it doesn't have. Here's how. During normal training, the network has a shared trunk and two heads. One head predicts the class. The other predicts how much the image has been rotated: zero, ninety, one eighty or two seventy degrees. At test time, a new image arrives. You rotate copies of it, take a few gradient steps on the rotation task to adjust the shared trunk, and then classify with the class head.

Maya So the rotation puzzle is a way to get a training signal out of an unlabelled image.

Theo And if the test image is corrupted, say heavy noise the network never saw, practising the rotation puzzle on noisy images retunes the features towards that noise. There are two versions. In the standard one, the weights are reset after each image. In the online one, they carry over from image to image.

Maya The online one is the continual-learning flavour.

Theo And it's where the dramatic numbers are. On CIFAR-10 with severe Gaussian noise, the error rate was forty-nine point four percent for the jointly trained model without test-time updates, forty-five point six with standard test-time training, and twenty-five point eight online.

Maya Nearly halved. And on ImageNet?

Theo With the same kind of noise, accuracy went from about two percent to twenty-six.

Maya Huge. What's the catch?

Theo The authors give it to you. Those online numbers come from a whole stream of images with the same corruption, so the model keeps adapting to one consistent shift. On a subtler real shift, a newer CIFAR test set, the gain was zero point eight of a point, which they call "small relative to the performance drop". The rotation task has to make sense: for airplanes in the sky, the paper notes it's often impossible even for humans to tell if the image is rotated. And their theory gives a sufficient condition only in a simplified convex setting: roughly, the rotation task's gradient has to point the same way as the main task's. It isn't a guarantee for deep networks.

Maya There's also a philosophical twist. The paper notes that online test-time training can deliberately forget the original training distribution when it's no longer relevant.

Theo Which, after the first half of this episode, sounds like heresy.

Maya Or like realism. Four years later, the same first author, Yu Sun, with Xinhao Li, Karan Dalal and many others: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States." And this is the bridge to the fast-weight story. Their abstract says it plainly: make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning.

Theo So the memory of a recurrent layer isn't a vector any more. It's the weights of a small model.

Maya Exactly. As the layer reads each token, its little inner model takes a gradient step on a reconstruction task: from one projection of the token, predict another. Then the layer's output comes from the just-updated inner model. The projections that define the task are learned in the normal outer training loop. So the network learns, during training, what kind of learning to do at test time.

Theo Learning to learn at test time. Hence the brackets in the title.

Maya And they show that in the simplest version, a linear inner model with batch gradient descent, this is equivalent to linear attention. Which closes the loop with Schlag's paper. The interesting versions go further: TTT-Linear, and TTT-MLP, where the inner model is a two-layer network.

Theo And does a cleverer memory actually buy you anything over the recurrent models people already use?

Maya On long contexts, in their tests, yes. They train models from a hundred and twenty-five million to one point three billion parameters, and compare with Transformers and Mamba, a strong recurrent architecture. The headline: as context gets longer, Mamba stops benefiting from more context after about sixteen thousand tokens, while the TTT layers keep reducing perplexity. At an eight-thousand-token context on the Pile, both TTT variants did significantly better than Mamba for the same training compute.

Theo Let me guess. It's slow.

Maya In wall-clock time, yes. The MLP version is far slower in practice than its compute count suggests, and the authors call their systems work "preliminary at best". They never trained on millions of tokens of context. And it's worth saying what it isn't: each sequence gets its own short-lived weights. When the sequence ends, they're gone. Nothing is learned permanently.

Theo Which brings us to the one that made headlines. Ali Behrouz, Peilin Zhong and Vahab Mirrokni at Google Research, posted on the last day of twenty twenty-four: "Titans: Learning to Memorize at Test Time."

Maya The one with "surprise".

Theo The surprise one. Same core idea: a small neural memory, whose weights are trained during the sequence to map keys to values. The gradient of that memory's error is what they call "momentary surprise". Big error, big surprise, big update. Then two additions. Momentum, so a surprising event keeps influencing updates for a while; they write that an event might not consistently surprise us over a long period although it is memorable. And a learned forgetting gate, which decays old memory so it doesn't fill up.

Maya And then they bolt it onto attention.

Theo Three ways. Memory as context, memory as a gate, memory as a layer. Short-window attention handles the recent past precisely, and the neural memory compresses the distant past. Plus a set of persistent, learned parameters that stay fixed at test time.

Maya And I'm guessing the numbers are good, because this one got a lot of press.

Theo They're good. At seven hundred and sixty million parameters, the best Titans variant had the lowest WikiText perplexity in their table, eighteen point six, against twenty-five point two for a Transformer baseline. And on needle-in-a-haystack retrieval at sixteen thousand tokens, one variant scored ninety-five point two where TTT and DeltaNet scored zero.

Maya And the fine print?

Theo The language models were trained on four-thousand-token sequences, and the largest is seven hundred and sixty million parameters. The abstract claims it scales to more than two million tokens of context, but the paper's only numeric table stops at sixteen thousand; the longer results are plotted, not tabulated. And the authors call it a first version, with larger models promised later.

Maya So if I tell a Titans model something today, does it remember it tomorrow?

Theo Not in anything this paper tests. Like the TTT layers, it's memory within a sequence. Clear the context and the memory's state goes with it. Which is exactly where the last part of our list picks up: what does it take for a model's real, durable weights to keep learning?

The LLM era: continual pre-training and SEAL 43:33

Maya Let's talk about real models and real weights. You've trained a large language model on hundreds of billions of tokens. A big new dataset arrives. Do you start again from scratch on everything, or keep training the model you have?

Theo Starting again is the safe answer, and the eye-wateringly expensive one.

Maya Adam Ibrahim, Benjamin Thérien, Kshitij Gupta and colleagues asked exactly that, in "Simple and Scalable Strategies to Continually Pre-train Large Language Models", published in Transactions on Machine Learning Research in twenty twenty-four. And they frame the two ways it goes wrong. Poor adaptation: the model fails to learn the new data. Or catastrophic forgetting: it learns the new data and loses capability on the old.

Theo McCloskey and Cohen, at the scale of the internet.

Maya And the first culprit is mundane: the learning rate. Pre-training ends with the learning rate decayed to a small value. If you just keep going at that tiny rate, the model barely adapts. So they rewarm it, raising the learning rate back up, then decay it again over the new data.

Theo Which helps it learn the new data, and smashes the old.

Maya Exactly. Rewarming on its own improved adaptation substantially, but, quote, "these models experience significant forgetting". Even rewarming on the same old data, with no change of distribution, temporarily raised its loss. Restarting the learning rate has a cost by itself.

Theo So the fix is replay. Of course it is.

Maya Of course it is. Mix some of the old data into every batch, swapping it for new tokens, so the total compute stays the same. They test two kinds of shift on GPT-NeoX style models: from the Pile to SlimPajama, a fairly similar English corpus, and from the Pile to German web text, a big shift.

Theo And how much old data does it take?

Maya Less than you'd fear. For the German shift with a four-hundred-million-parameter model, no replay gave an average validation loss of two point three four. With twenty-five percent replay, one point seven five. A model trained from scratch on both datasets together: also one point seven five. Matched. And at ten billion parameters, on the English shift, five percent replay reached an average loss of one point eight nine, against one point eight seven for training from scratch on everything.

Theo That's a striking result. Almost the same model, without paying for a full retraining.

Maya With caveats they mostly state themselves. One run per setting, no multiple seeds. Mostly one transition, not a long chain. And when you look at individual benchmarks, averages hide things. At ten billion parameters the average benchmark score was forty-seven point seven against forty-eight point zero, nearly identical. But on MMLU specifically, the continually trained model scored twenty-eight point eight versus thirty-seven point eight.

Theo A nine-point gap hiding in a matched average.

Maya The authors flag it and suspect the limited amount of training data, without establishing the cause. And they note that a later group, Glorioso and colleagues, applied the same recipe without any MMLU drop. Separately, the compute saving assumes you already have the pretrained model, and that you still have the old data to replay.

Theo Now from the most practical paper on our list to the most science-fiction-sounding one. Adam Zweiger, Jyothish Pari and colleagues at MIT, NeurIPS twenty twenty-five: "Self-Adapting Language Models". SEAL.

Maya The model that writes its own homework.

Theo That's about right. When the model gets new information, say a passage of text, it doesn't just train on the passage. It writes a "self-edit": its own version of what to learn. For a passage, that might be a list of implications and restatements of the facts. The model is fine-tuned on its self-edit, with a small LoRA adapter, then tested on questions about the passage with the passage removed. If the score goes up, that self-edit gets rewarded. A reinforcement learning loop trains the model to write better self-edits.

Maya Their analogy is students turning a lecture into revision notes.

Theo Exactly theirs. The results: on single SQuAD passages, Qwen two point five, the seven-billion-parameter version, answering without the passage in front of it. The base model scored thirty-two point seven percent. Fine-tuning on the raw passage, thirty-three point five, so verbatim training barely helps. Fine-tuning on synthetic notes from GPT-4.1, forty-six point three. On SEAL's own trained self-edits: forty-seven point zero.

Maya So a seven-billion-parameter model writing better study notes for itself than GPT-4.1 writes for it.

Theo In that single-passage setting, yes. But in the larger pooled test, with two thousand and sixty-seven passages, it reverses: GPT-4.1's notes scored forty-nine point two and SEAL's forty-six point four. And the appendix has a very honest table. Just changing the prompt, asking the base model to "rewrite" the passage, with no reinforcement learning at all, reached forty-nine point four.

Maya So a lot of the gain is in the format of the notes, not the learning loop.

Theo A lot of it, though reinforcement learning with that rewrite prompt pushed it to fifty-five point six. The bigger point is in their limitations section. When they apply edits one after another, a stream of passages, "performance on earlier tasks gradually declines as the number of edits increases".

Maya It forgets.

Theo It learns how to learn, and it still forgets. Every self-edit also needs a fine-tuning run and an evaluation, thirty to forty-five seconds each, and rewards need known answers to test against. So it isn't yet a model that learns autonomously from whatever it reads.

Maya Which brings us to the paper that, to me, has the most interesting answer to that forgetting.

Sparse memory fine-tuning and Nested Learning 50:16

Theo Jessy Lin, Luke Zettlemoyer, Gargi Ghosh and colleagues at Meta's FAIR lab, with Berkeley, October twenty twenty-five: "Continual Learning via Sparse Memory Finetuning". And their diagnosis goes right back to McCloskey and Cohen. Forgetting is hard to stop because every task shares the same trainable parameters.

Maya So, stop sharing them.

Theo That's the idea, made practical with an unusual architecture. They take a language model and replace one of its feed-forward layers with a memory layer: a giant lookup table with a million slots, each holding a learned key and a learned value. For every token, the model forms a query, picks the thirty-two best-matching keys, and blends their values.

Maya So for any given token, only a tiny part of the table is used.

Theo Tiny. They point out that's about thirty-two thousand active parameters in that layer, instead of fifty million in the feed-forward layer it replaces. Now the clever part. When you want to teach the model something new, you don't update everything it touched. For each training batch, you count which slots were used. Then you compare against how often those slots are used on generic background text. Slots that are busy everywhere probably do general work, like grammar. Slots that are unusually busy for this batch are probably about this fact. You update only the values in the top-ranked slots: five hundred of them, for a single fact.

Maya That's TF-IDF. The old search-engine trick.

Theo Exactly that. Term frequency times inverse document frequency, applied to memory slots. Slots used heavily by this batch but rarely in general get written. Everything else is frozen for this update.

Maya Does it work?

Theo In their setting, strikingly. They teach a one point three billion parameter model a sequence of facts from TriviaQA, one at a time, then measure how much it degrades on a different benchmark, NaturalQuestions. Full fine-tuning: that score dropped by eighty-nine percent. LoRA: seventy-one percent. Sparse memory fine-tuning: eleven percent. For the same level of learning the new facts.

Maya Eighty-nine versus eleven.

Theo And a nice ablation: rank slots only by how often this batch uses them, without the background comparison, and you learn as much but forget more. The background contrast is doing real work. Their example of a fact: "What was the name of the cat in Rising Damp? Vienna." Of a hundred and sixty-nine slots in that fact's core set, twenty-five were needed to answer.

Maya A British sitcom cat. I approve.

Theo Now the caveats. It's a preprint. The tests are factual question answering: facts and document chunks, not skills or reasoning. The memory model has a different architecture from the baselines, and started out better at the questions. The baselines used AdamW and sparse fine-tuning used plain SGD, though they report that it still forgot less when every method used AdamW. And it's one layer, one model size. The authors themselves say it needs to scale to harder tasks and larger models.

Maya But conceptually it's the cleanest answer we've seen to the nineteen eighty-nine geometry. If new learning overwrites old learning because they share weights, give each new fact a place of its own to be written.

Theo Sparsity as isolation. Progressive networks did it with whole columns. This does it with a handful of table entries.

Maya Our last paper is the most ambitious, and the hardest to summarise fairly. Ali Behrouz, Meisam Razaviyayn, Peilin Zhong and Vahab Mirrokni: "Nested Learning: The Illusion of Deep Learning Architecture". A version appeared at NeurIPS twenty twenty-five, and Google promoted it as a new paradigm for continual learning.

Theo And it opens with an amnesia analogy, which is a lovely echo of our cold open.

Maya It is. McCloskey and Cohen said their network's forgetting looked like retrograde amnesia: losing the past. Nested Learning says today's large language models have anterograde amnesia: a condition where you can't form new long-term memories while old ones stay intact. Their knowledge is either in the context window, or in weights frozen at the end of pre-training. They experience "the immediate present as if it were always new".

Theo Thirty-six years, and the two analogies point in opposite directions. That's the whole dilemma. Update the weights carelessly and you lose the past. Freeze them and nothing you tell the model today becomes a lasting memory.

Maya Their proposal is a way of looking at models. Every part of a learning system, they argue, is an optimisation problem with its own stream of data and its own update rate. The optimiser's momentum is a little memory that learns from gradients. Attention is a memory that learns from the context. The pre-trained weights are a very slow memory. So instead of architecture plus optimiser, you get nested learners running at different speeds.

Theo Fast weights, slow weights, and everything in between.

Maya And they build on that. A "continuum memory system": feed-forward blocks that update on different schedules, some every few tokens, some much less often. Combine it with a self-modifying version of the Titans memory, and you get an architecture they call Hope.

Theo Go on, how does Hope do? And I want the retrieval number, not just perplexity.

Maya Fair. First the good news. At one point three billion parameters, trained on a hundred billion tokens, Hope's average across their reasoning benchmarks was fifty-eight, against fifty-three for a Transformer baseline, with lower perplexity too. And removing the continuum memory made it worse.

Theo And retrieval?

Maya On a sixteen-thousand-token test that hides several keys in a long document, Hope scored fourteen point eight. A full Transformer scored sixty-one. So it beats the other attention-free models, not attention. Though their own variant that swaps standard attention back in, called Hope-Attention, gets back to about sixty. Their continual-learning classification results are shown as plots, so I can't quote a number. And for adapting real Llama models, the comparison bundles several changes together.

Theo And yet, credit where due, the paper has a section titled "Is Catastrophic Forgetting Solved?"

Maya And the answer, in their words, is that catastrophic forgetting is "not 'solved' in general". They call Nested Learning "a roadmap rather than a destination". And one more line I find clarifying: they say test-time training and test-time memorisation are "instances of parametric in-context learning", where the acquired knowledge doesn't persist once the context is removed.

Theo Which is the distinction we've been circling all episode. Weights that change while you read are working memory. Weights that change and stay changed are learning.

Four questions to ask any continual-learning claim 57:59

Maya Let's pull it together. Twenty papers, thirty-six years. What does the field actually know?

Theo I'd start with what it doesn't. Nobody has a general solution to forgetting. Not the nineteen eighty-nine authors, not the twenty twenty-five ones, and the honest papers say so.

Maya Agreed. But it does know a lot about how not to fool yourself. We've ended up with four questions worth keeping beside any continual learning headline.

Theo Go on.

Maya One: what help did the model get? At test time, was it told which task it was solving? That's van de Ven and Tolias: the same method went from ninety-eight percent to twenty just by taking that label away. And during training, was it told when the task changed? Those are different kinds of help, and they change what an experiment shows.

Theo Two: where does the memory live? In a buffer of old examples, like iCaRL and the tiny memories. In protected or isolated weights, like EWC and the sparse memory slots. Or in fast weights that exist only during the current sequence, like TTT layers and Titans. That last kind vanishes when the context clears. It's working memory, however clever it is.

Maya Three: can it still learn? Dohare and colleagues showed that a network can avoid forgetting and still slowly lose the ability to learn. If a paper only measures retention over five or twenty tasks, it hasn't tested the second problem.

Theo And four: what did it cost, and what was it compared with? Fifty epochs per task, or one pass. A baseline curve copied from another paper. An average that hides a nine-point drop on one benchmark. None of these make a result wrong, but they all change what it means.

Maya And with those four questions, what actually works today? Replay, first: the least glamorous idea in the field, and it keeps winning, from one stored example per class up to a quarter of every batch in continual pre-training. Then isolation, done properly: whole columns in progressive networks, or a few hundred memory slots in sparse fine-tuning, which I'd watch most closely, precisely because so far it's only been shown on facts. And keeping the network plastic: small weights, and now and then a fresh unit in place of a little-used one.

Theo And the frontier is where those threads cross. Titans and Hope learn quickly within a context. SEAL decides what to learn from new material. The missing piece, which none of them demonstrates, is reliably turning that fast learning into durable weights without breaking what's already there.

Maya Consolidation. Moving things from working memory into long-term memory. Retrograde amnesia in nineteen eighty-nine, anterograde in twenty twenty-five. Consolidation is what would sit between them.

Theo And it's exactly what the brain-inspired papers point to and none of them prove. The Nested Learning paper says outright that it focuses only on the online part, not the sleep-like stage.

Maya Two papers to read, if you only read two?

Theo Van de Ven and Tolias, "Three scenarios for continual learning", because it teaches you to read every other paper. And Dohare and colleagues in Nature, "Loss of plasticity in deep continual learning", because it's the half of the problem most people still ignore.

Maya And if you want a third, for where things may go: Lin and colleagues on sparse memory fine-tuning.

Theo A transparency note, as always. This episode was written by an AI assistant, working from the full text of all twenty papers, and voiced with synthetic speech. Where a result was small, simulated, a preprint, or only shown in a figure, we've tried to say so. But check the papers yourself.

Maya And there's an irony worth naming. The assistant that wrote this has no weights that learn from its conversations. Its "memory" is a database of notes, looked up and placed into its context each time. In this episode's terms, that's external storage feeding working memory. Not replay training, and no consolidation at all.

Theo So the show about weights that keep learning was written by a system whose weights didn't move once.

Maya Which, after twenty papers, looks less like a cop-out and more like the honest state of the art: keep good notes, and reread them.

Theo Thanks for listening to The Reward Signal. I'm Theo.

Maya I'm Maya. Keep learning. Try not to forget.