A frozen Gemma 3 checkpoint can get meaningfully sharper without a full retrain. Google DeepMind's recirculation paper reports about a 23% drop in perplexity and about a 21% lift on GSM8K when you add a specific kind of inference-time recurrence. The base weights stay frozen. You change how activations flow during the prefill pass.
That split matters if you run local Gemma for evals, agents, or privacy-sensitive prototypes. You might get a better model from the same GGUF file, but only if your inference stack can implement the hook.
What recirculation actually does
Transformers process a prompt in parallel during prefill. That parallelism is great for speed. It is awkward for state tracking.
DeepMind frames the failure mode with a fishing-at-the-river-bank dialog. The model resolves the word "bank" in deep layers, then answers an ATM question using shallow layers that still see the ambiguous embedding. Lepori et al. showed that patching deep activations into shallow layers cuts those contextualization errors sharply. Recirculation generalizes that idea into a full inference recipe.
Instead of looping the same layer block (the popular "looped transformer" trick), recirculation leaks a convex mixture of a deep layer's residual stream into a shallower layer after each input token. The model runs two parallel stacks per recurrence step. State can live in the same depth across time, which is what you want for belief tracking.
The paper stresses this is not chain-of-thought. CoT is for hard reasoning steps. Recirculation is for keeping a coherent internal state while the model reads.
Prefill cost vs flat generation
Here is the trade teams should budget for.
| Phase | Baseline transformer | With recirculation |
|---|---|---|
| Prefill (read the prompt) | Parallel across tokens | Serial token updates, extra stack pass |
| Generation (decode new tokens) | Standard autoregressive | "Essentially no additional latency" per the paper |
| Weight changes | None for basic variant | None for adaptive (small MLP only) |
Modern accelerators parallelize the dual-stack work well once you are decoding. The pain shows up when you ingest a long document, a fat tool schema, or a whole repo summary in one shot. Your time-to-first-token can climb even if per-token generation feels normal.
I would not dismiss the method because prefill is slower. Many agent loops are decode-heavy after a chunky first context. But if your product is "paste 200K tokens and answer in two seconds," recirculation is the wrong default until hardware or kernels catch up.
Numbers on Gemma 3
The authors sweep source and destination layer pairs on Gemma 3 pretrained sizes (1B, 4B, 12B). Optimal pairs differ by scale. Examples from the tuning set: layers 11→4 on 1B, 18→9 on 4B, 35→16 on 12B.
Basic recirculation (fixed mixture, no weight updates) already cuts perplexity on most eval sets. Gains grow on longer documents. Short Lambada-style snippets barely move.
Adaptive recirculation trains only an MLP that outputs per-token mixture vectors α and β while the Gemma weights stay frozen. On Gemma 3 1B that variant hits a 23.0% mean perplexity reduction across nine datasets, beating full fine-tuning of the augmented architecture in their setup.
Downstream tasks are mixed on tiny multiple-choice benches. The standout is GSM8K math with chain-of-thought prompting on Gemma 3 4B pretrained:
| Metric | What improved | Adaptive recirculation |
|---|---|---|
| pass@1 (greedy) | Sharper best answer | 8.8% error reduction |
| pass@128 (sampled) | More correct candidates in the pool | 20.9% error reduction |
The abstract rounds that story to about a 21% accuracy increase on GSM8K, which matches the pass@128 story where recirculation expands what the model can find, not just what greedy decoding picks.
Recirculation also beats naive looping on Gemma 3 in their heatmaps. Looping pretrained weights did not show robust gains at smaller scales. Same family, different mechanism.
Frozen weights, light tuning
Two deployment modes matter for builders.
- Training-free recirculation. Pick layers and α from a grid search on a small text slice. Ship as an inference flag. No checkpoint swap.
- Adaptive recirculation. Freeze Gemma. Train the mixing MLP on a few hundred short documents (arXiv, C4, PG-19 slices in the paper). Gains jump, but the MLP training data matters. They warn that tuning on the wrong benchmark can hurt generalization.
Neither path touches the billion-parameter matrix you already downloaded. That is the practical win. You avoid a fragile "fork the whole model" deploy when you just want better state tracking on instruction following or math.
The paper also notes recirculation works on other families (Pythia, Qwen3, Phi-2, Ministral3) with smaller effect sizes until you tune normalization. Gemma 3 is unusually receptive.
If you run local Gemma today
Most laptop and edge stacks in August 2026 still assume a flat transformer forward pass. Recirculation needs:
- A hook between specific layer indices during prefill only (or a faithful reimplementation of their serial unrolling)
- Hyperparameters per model size (do not copy 4B layer ids onto 12B)
- Benchmarks on your context lengths, not just GSM8K
What I would do on a client eval before betting production routing on this:
- Reproduce perplexity on a 2k and 32k slice of your real prompts, not arXiv abstracts.
- Run your agent harness with and without recirculation on 50 golden tasks.
- Measure prefill latency at your median and P95 context size.
- Keep a hosted fallback if prefill blows your SLA.
Ollama and llama.cpp ecosystems will likely absorb this if the paper holds up. Until then, treat recirculation as a research lever for teams that own their inference code, not a toggle in the model card.
Docs worth bookmarking alongside the paper:
Where this fits Models & tooling
This is searchable news with a builder angle: architecture hacks at inference time that do not require a new training run. For my pillar on models and tooling, the lesson is to separate "we need a bigger model" from "we need a better pass over the same weights."
Recirculation will not fix a bad quant or a model that fails your domain eval. It might tighten coherence on long prompts, polysemy, and math chains on Gemma 3 checkpoints you already trust.
If you are choosing between shipping another API route or squeezing more from local Gemma on privacy-sensitive workloads, I help teams map that trade with real latency and quality numbers. Book a free discovery call if you want a second opinion before you rewire your stack.