Every frontier model is frozen after pre-training. Continual learning is still mostly a research poster.
On June 24, 2026, Zyphra Research published plasticity loss results on GPT-style decoder-only transformers from 5M to 314M non-embedding parameters. Models trained on a multilingual continual learning setup eventually lose the ability to learn, tracked via deterioration on a held-out Vietnamese probing task.
AlphaSignal summarized it as "all LLMs eventually lose the ability to learn new things." The paper title asks the sharper question: Can Scale Save Us From Plasticity Loss in Large Language Models?
Short answer from Zyphra: bigger helps, but sublinearly. Scale delays the cliff. It does not remove it.
Plasticity vs catastrophic forgetting
Continual learning discussions often mix levels:
| Level | Problem | Status in 2026 |
|---|---|---|
| Session memory | Remember earlier in one chat | Mostly solved via long context |
| Task continual | Learn task B without erasing task A | Active research |
| Lifelong plasticity | Keep adapting after massive prior training | Hard open problem |
Plasticity loss is about the network losing adaptability even when you still have gradient updates. Not just forgetting one task. The weights become stiff.
Zyphra's setup uses real multilingual data at scale, not tiny MNIST-style toys. That matters because skeptics could dismiss old plasticity papers as irrelevant to LLMs.
The scaling law headline
Across every scale studied, evidence appeared that transformers eventually lose plasticity under continual training.
Zyphra fit a scaling law for when loss of plasticity begins as parameter count grows. Key implication:
- Onset delay grows sublinearly with parameters
- Throwing 10x params might buy less than 10x delay
- Parameter scaling alone is likely insufficient as the sole fix
For teams betting "our 70B will just keep learning from logs," this is a caution sign. Stationary training showed plasticity loss too, not only abrupt task switches. Long pre-training runs may hit the same wall.
Why agent builders should care
Agents need online adaptation: new APIs, new client vocab, new compliance rules. If base models lose plasticity during extended fine-tuning, your agent memory layer (RAG, skills files, vaults) becomes more important, not less.
This connects to:
- LLM sleep and offline consolidation research on alternative update schedules
- Metis native memory foundation models betting on external memory instead of weight updates
- Client stacks where we never rely on continual weight training in prod
Zyphra's Zyphra FM line (open-weight models) positions the lab as serious about training dynamics, not only inference speed.
What the field might do instead of naive scale
The paper points toward interventions beyond width:
- Optimizers and regularizers that preserve plasticity (active research area in older nets, now LLM-scale)
- Modular adapters instead of full-weight continual updates
- Retrieval and tool memory as the primary adaptation surface
- Periodic partial resets or replay mixes (engineering hacks with their own cost)
None of these are productized guarantees yet. The research says default training is not continual-learning ready.
Limits on extrapolation to 70B+
Zyphra studied up to 314M non-embedding parameters. Frontier models are orders of magnitude larger. Sublinear scaling suggests delay, not immunity, but direct measurement at 70B is still needed.
Probing task is Vietnamese-specific. Mechanism may vary by language and domain.
Bottom line
Zyphra's June 2026 paper is a sober check on continual learning hype. GPT-style transformers lose plasticity at every tested scale. Scale postpones the problem on a sublinear curve. For applied AI, that reinforces a design pattern I already ship: adapt through memory, tools, and retrieval, not assume the base model will keep absorbing new client reality forever.
Evaluating long-horizon agent memory architecture? Book a free call.