Why We May Never ‘Solve’ Continual Learning

Continual learning,

It’s a defining but poorly understood feature of biological intelligence. In artificial systems, we do not yet know what it would take to reproduce this capability, if it's even possible to replicate in a biologically-similar way, or how essential it is to further AI progress.

In the end, it might turn out that there was one clear path to solving continual learning. Maybe that path will lead us to AGI and beyond. And maybe we’re on that path now.

It's also possible that there will be many paths to progress.

Or maybe the current training paradigms and architectures will simply dissolve into something else entirely?

This essay is my attempt to make better sense of the challenge.

what is continual learning?

Continual learning is the ability to acquire, retain, update and reuse knowledge across an ongoing stream of experience.

In your biological brain, you absorb information from experience. That new information is shaped by what you already know. Somehow, you protect that old knowledge from being overridden by the new.

New experience has to be integrated without overwriting what the system already knows. Editorial illustration from The AGI Post video.

We do not yet have a full explanation for how, why or where this information is stored. We know there is stuff happening in your hippocampus and neocortex, and there are some theories related to evolutionary pressures etc, but we don’t fully understand it.

The truth is - we don’t know fully what’s going on here.

Think about this for a moment, it is an underappreciated thing - whatever is happening here is so fucking strange and remarkable! Grokking this fact, that compression and integration and recall of information is a really remarkable piece of biological machinery, helps us understand continual learning, the nature of intelligence, and the magnitude of solving this problem.

Now the kind of AI you and I spend most of our time with - it doesn’t work this way.

Language models are trained with HUGE amounts of compute and data, and their brains are essentially frozen until the next training cycle. They do not learn directly from their experiences or interactions.

But earlier, pre-transformer neural networks and deep learning kinda did. And this is an important part of the story most people leave out.

Before we get into this, I want to quickly share the implications of continual learning.

Yesterday, Demis shared his framework for frontier AI. In it, he said.

“

Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.

he also said.

“

The magnitude of this technology’s impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed.

A couple of weeks ago, DeepMind published an AGI to ASI paper,

in it, they specifically called out continual learning as a bottleneck to AGI.

“

an AGI-level system would posses the capability for continual learning to perpetually accrue competence from interactions without catastrophic forgetting (Kirkpatrick et al., 2017; Wang et al., 2021).

When Demis or Dario or Illya talk about ‘continual learning’, we can assume they’re talking about something that can learn directly from its own experiences, and use those learnings to adapt future action, without requiring a completely new training cycle.

One more point I’ll make here, is that continual learning is at the heart of the current open versus closed source model debate in a really interesting way - many neolabs and projects are making the bet that enterprise and individuals will make more progress toward continual learning only if they have the ability to collect and curate their own data, and feed that data back into a model where they have access to the weights.

To appreciate the complexity of the challenge, and to better understand how and when we actually make progress, it's helpful to understand a brief history of continual learning and how the problem has changed over time.

This might be obvious if you’ve studied the history of deep learning. But if you haven't, it's an important piece of the puzzle.

Continual learning is not a new problem, it has just changed shape.

Back in the 80’s, researchers studying early neural networks noticed that learning new patterns would severely disrupt previously learned patterns. This became known as catastrophic forgetting.

They trained small neural networks on a set of simple mathematical facts, then on a new set of facts, and observed that the model would forget much of what it learned from the initial set.

Deep learning made this problem harder.

A deep neural network learns representations and patterns. It doesn’t store knowledge in neat, separate boxes, rather in abstract representations (much like a biological brain).

In a 2017 deepmind paper titled - "Overcoming catastrophic forgetting in neural networks," it introduced the Elastic Weight Consolidation (EWC). An algorithm which would allow the network to update its weights without losing previously learned knowledge.

I’m going to give a very crude and condensed example so you get the gist.

They basically taught a system how to play a bunch of Atari games.

They first taught the system to play pong - detect a fast-moving ball, move the paddle up and down etc. Then they taught the model to play space invaders - detect multiple rows of aliens, a shield barrier, and incoming lasers. Move left, move right, fire.

The problem - when the network learned how to shoot aliens for space invaders, it forgot how to move its pong paddles.

To prevent this from happening - the EWC algorithm helped identify and protect the learned patterns responsible for high pong scores - forcing it to adjust its new learnings around the existing knowledge.

Elastic Weight Consolidation protects parameters important to task A while finding a route toward task B. Source: Kirkpatrick et al., 2017, Figure 1.

This was repeated across ten different Atari games.

Large Language Models are just one specific type of deep learning. Unlike some older networks designed to learn continuously while interacting with their environment, Language Models do not do this.

When a lab builds a base model, they do pre-training with huge amounts of data and compute - they are teaching an algorithm to get progressively better at predicting the next token.

So you have this frozen brain with a huge amount of knowledge, but it still lacks the ability to follow instructions, maintain back and forth conversation, use tools or produce safe and aligned responses.

If you were interacting with the base model, it would be incoherent.

To fix this, we unfreeze the brain and do further distinct ‘post-training’ work - instruction tuning and reinforcement learning further adjusting the weights.

During this post-training phase the original catastrophic forgetting challenge returns - It’s a huge engineering challenge to run these post-training pipelines at scale while balancing alignment and efficiency. A known problem with LLMs - is that optimizing heavily for safety or conversational style can cause the model to "forget" how to code and use tools efficiently.

Im going to take a pause here, because this is where things get slightly confusing - we have this old continual learning challenge with older neural networks. Which was - how do we allow a neural network to learn to play space invaders without forgetting how to play pong.

And that same problem now exists in the post-training pipeline for modern LLM's. How do we get a pre-trained base model to be a good and safe chatbot, good at writing code and essays, without losing any of its existing knowledge.

The InstructGPT post-training pipeline: supervised fine-tuning, preference collection and reinforcement learning. Source: Ouyang et al., 2022, Figure 2.

see how the challenge has just evolved?

DeepSeek-V3 base was trained on 14 trillion tokens. This base model was then used to create the R1 reasoning model through a complex sequence of SFT and RL on math and code.

Side note - when markets opened on the Monday following the DeepSeek-R1 release, Wall Street melted. Over $1 trillion wiped from U.S. stock market in a few hours.

If you’re interested in the technicals - the DeepSeek-V4 paper introduces some really interesting architectural changes to address this same challenge using a parallel training process and distillation.

The current paradigm has taken us a surprisingly long way - much farther than most expected. Late last year, there was a palpable step-change in coding agent capability. This takeoff was likely enabled by intensified post-training on code, reasoning, tool use in high verifiability/economically valuable domains.

No question there’s more juice to be squeezed.

However, its an open question as to how far we can take this, and when we actually hit diminishing returns. there are so many potential variables pushing the scales in either direction.

The open question is whether post-deployment data could fuel a new paradigm of model capability improvements.

By post-deployment, we just mean when the model is set out into the wild. There are hundreds of millions of people using these things every day. The labs are already using deployment data to train the next model family, but the models aren’t learning from these experiences and adapting within their deployment cycle.

How this deployment data should be used is still a huge and open debate.

Giving a system autonomy to modify its own external scaffolding - skills, memory etc could be considered continual learning (depending on your definition). Harness engineering, Loop Engineering, OpenClaw, and most other harnesses are doing exactly this.

Even without modifying the weights, a model is able to act > observe consequences > update, and then act differently based on the learnings, and ultimately encounter and compound better experience as a result.

An agent can act, observe environmental feedback and act again while its base weights remain unchanged. Source: Anthropic, 2024.

But the continual learning imagined by Dwarkesh and others involves the model updating its weights.

This was a quote from the end of his article.

“

For what it’s worth, I’m not talking about the upcoming dreaming feature in the leaked Claude Code source code, which I’m guessing will be more about the model writing lots of Markdown files for itself. I mean actually updating the weights themselves. I just don’t think you can accumulate new skills by passing yourself notes.

He uses an instrument analogy to explain.

“

imagine if the way students learned how to play the saxophone is by giving this new instrument you’d never tried before a go, taking some notes about what went wrong, and then giving it to the next student who is also playing it for the first time

I think there is a deeper thread here, which is that compressing and integrating information back into the existing weights is where we’re hoping to find the real juju. How or if this will work in practice under the current training regime is still an open question.

Engram are doing really interesting work - they build and train bespoke models and adapters for companies.

The idea is that some company information should be written back into the weights - in the same way you take notes, but also internalise and intuit information back into your meatsack brain.

They build custom models to more deeply understand the organisation - more like a long-term employee, than an outsider looking in.

They’re using a bunch of different methods in combination to do this.

ways to extend and adapt and nudge the model weights - TLDR

• Lightweight adapters - LoRAs, trained prefixes. train a small collection of additional parameters for specific companies, or teams within companies. Those parameters don’t necessarily change the underlying weights, they just nudge the behaviour of the base model

• Cartriges - compress large bodies of company information into reusable internal context the model can load when needed.

• Sparse trainable memory - sparse memory fine-tuning take an open model like Qwen, and customise the architecture of that model, replacing some of the normal layers, with these memory layers. into which you can tune specific company information without effecting the entire model.

I assume they’re also doing full parameter SFT and RL, and also distillation, depending on the use case.

This is enterprise, but it also applies to the individual use case.

Thinking machines are also doing really interesting work. founded by Mira Murati (former CTO of OpenAI), they diverge from the other major labs in that they are pursuing this more continuous, collaborative, human-aligned ai.

In a recent example - they were working with Bridgewater, one of the world’s largest investment firms to see whether they could ‘teach’ a model to replicate the judgement of experienced investors (call this intuition?).

As would be the case with most expert knowledge - investors could often recognise important insight and patterns, but not fully explain the process.

They collected examples of the investors’ actual decisions and trained Qwen (an open model) on the data. Instead of telling the model how an expert thinks, they showed it thousands of examples of expert judgment and updated the weights until it began making similar decisions.

The final customised Qwen hit 84.7% accuracy, beating every frontier model they tested (at a fraction of the cost).

On six information-filtering tasks, the customised Qwen-based model reached 84.66% average accuracy at $4.72 per 1,000 tasks. Source: Bridgewater AIA Labs and Thinking Machines Lab, 2026.

As I was writing this, Thinking Machines dropped Inkling. A large open-weight model specifically designed to be customised through its Tinker training platform. In one demonstration, Inkling generated its own training data, fine-tuned its weights, evaluated the result and switched itself to the improved checkpoint.

“

To show what customization means in practice, we asked Inkling to fine-tune itself. Using Tinker, the model wrote its own fine-tuning job, ran it, and evaluated the result:

Inkling is an open-weights model designed to be customised through Thinking Machines Lab’s Tinker platform. Source: Thinking Machines Lab, 2026.

You can see the bridge to continual learning.

Which begs the question - is the path to true continual learning every company and every individual owning their own continual learning pipeline?

This is not to say the labs won’t pursue a different kind of continual learning, leveraging deployment data in a different way.

it’s worth pointing out that there are some very smart people who believe the current Language Model/Transformer paradigm may not be the best foundation from which to build the most efficient incarnation of continual learning.

The open question is whether LM’s remain the foundational layer, or become one component in a broader system with a completely different overall architecture.

David Silver led the development of AlphaGo and AlphaZero. Last year, with Richard Sutton, he published The Era of Experience.

The Era of Experience

Their argument is that progress from training on static human-generated data will eventually run into a wall. To move significantly further, AI systems will need to generate a new source of data by acting in the world, observing the consequences and continually learning from their own experience.

David Silver and Richard Sutton argue for agents that learn continually from streams of experience. Source: The Era of Experience, 2025.

Silver has now left DeepMind and founded Ineffable Intelligence to pursue this idea: systems that do not just imitate the knowledge humans have already produced, but discover new knowledge for themselves through reinforcement learning and experience.

Others, like Yann LeCun have been more vocally critical of the current training paradigm, insisting we should rethink the architecture.

Which leads me with the thought that there may not be a single path to solving continual learning. The shape of challenge may continue evolving in many different directions. Will be very interesting to see how all this plays out.

Continual learning, memory and catastrophic forgetting

• Abel, David, et al. “A Definition of Continual Reinforcement Learning.” NeurIPS, 2023.
• Squire, Larry R., et al. “Memory Consolidation.” Cold Spring Harbor Perspectives in Biology, 2015.
• Patel, Dwarkesh. “Adam Marblestone — AI Is Missing Something Fundamental About the Brain.” Interview, 2025.
• McCloskey, Michael, and Neal J. Cohen. “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem.” Psychology of Learning and Motivation, 1989.
• Kirkpatrick, James, et al. “Overcoming Catastrophic Forgetting in Neural Networks.” Proceedings of the National Academy of Sciences, 2017.
• Wang, Dequan, et al. “Tent: Fully Test-Time Adaptation by Entropy Minimization.” ICLR, 2021.

Frontier models and post-training

• Vaswani, Ashish, et al. “Attention Is All You Need.” NeurIPS, 2017.
• Ouyang, Long, et al. “Training Language Models to Follow Instructions with Human Feedback.” 2022.
• Luo, Yun, et al. “An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning.” 2023; revised 2025.
• DeepSeek-AI. “DeepSeek-V3 Technical Report.” 2024; “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” 2025; and “DeepSeek-V4 Technical Report.” 2026.
• Reuters. “US Tech Shares Recover Some Losses from Steep DeepSeek Selloff.” 28 January 2025.
• Karpathy, Andrej. “Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI.” No Priors, 2026.

AGI, deployment learning and model ownership

• Hassabis, Demis. “A Framework for Frontier AI and the Dawning of a New Age.” 14 July 2026.
• Genewein, Tim, et al. “From AGI to ASI.” Google DeepMind, 2026.
• Amodei, Dario. “AGI Might Not Need Continual Learning.” Interview, 2026.
• Sutskever, Ilya. “We’re Moving from the Age of Scaling to the Age of Research.” Interview, 2026.
• Patel, Dwarkesh. “The Next Big Breakthrough Will Be AIs Learning on the Job.” 26 June 2026; see also “What Does the Next Training Paradigm Look Like?”
• Anthropic. “Building Effective AI Agents.” 2024.
• Nadella, Satya. “The Reverse Information Paradox.” 12 July 2026.

Engram and Thinking Machines

• Sequoia Capital. “Memory and Continual Learning: Engram’s Dan Biderman and Jessy Lin.” 24 June 2026.
• Catanzaro, Sarah. “Announcing Our Investment in Engram, the Memory Dream Team.” Amplify Partners, 23 June 2026.
• Su, Sarah, et al. “Learning to Replicate Expert Judgment in Financial Tasks.” Bridgewater AIA Labs and Thinking Machines Lab, 30 June 2026.
• Thinking Machines Lab. “Inkling: Our Open-Weights Model.” 15 July 2026.

Architectures beyond the current paradigm

• Silver, David, and Richard S. Sutton. “The Era of Experience.” 2025.
• Ineffable Intelligence. Official website and mission statement. 2026.
• LeCun, Yann. “A Path Towards Autonomous Machine Intelligence.” 2022; see also Meta AI’s “Yann LeCun on a Vision to Make AI Systems Learn and Reason Like Animals and Humans.” 2022.