Large language models eat training data orders of magnitude larger than what biological learners need. Everyone knows the scaling laws. Fewer people ask why next-token prediction is so hungry.
A June 2026 paper from Inquiring Lines, "Learn from your own latents and not from tokens: A sample-complexity theory", puts a number on the gap.
For hierarchical compositional data, token-level learning needs exponential samples in hierarchy depth. Latent prediction needs constant samples. Same data generating process. Different objective. Exponential vs constant.
That is not a marginal improvement. That is a different scaling regime.
The setup: Random Hierarchy Model
The authors use a tractable probabilistic context-free grammar (the Random Hierarchy Model, RHM) that captures compositional structure in language and images.
The grammar generates visible token strings by recursively applying production rules along a tree of hidden symbols at depth L. Think: characters form words, words form phrases, phrases form sentences, all with latent structure you never observe directly.
The question: how many samples do you need to recover that hidden tree?
| Training objective | Sample complexity (depth L) |
|---|---|
| Supervised / token-level SSL | Exponential in L (~m^(L+1)) |
| Latent prediction (self-predicting) | Constant in L (~m^3, up to log factors) |
The constant-vs-exponential split is the headline. At depth L=10, that is not a 10x gap. It is a qualitatively different learning curve.
Why latents beat tokens
The mechanism is correlation structure.
Latents at the same level of the hierarchy are far more correlated with each other than with raw tokens. When you predict from your own latents, you amplify a signal that token-level prediction dilutes.
Token prediction asks: given these characters, predict the next character. The model must infer hierarchy from surface statistics alone.
Latent prediction asks: given this internal representation of a related view, predict your latent representation of another view. Same-level latents share structure tokens hide.
This connects to predictive-coding accounts of cortex and to methods like:
- JEPA (Joint Embedding Predictive Architecture)
- data2vec (predict latent targets from masked views)
- H-JEPA and multi-scale variants
The paper gives the first sample-complexity analysis of data2vec and shows it implicitly performs hierarchical latent prediction. Explicit stacking (H-JEPA style) may be largely redundant.
Three confirmations in the paper
The theory is not hand-wavy. The authors confirm the bound three ways:
- Hierarchical clustering algorithm that recovers the latent tree with constant samples
- End-to-end neural network with predictor-clusterer modules that predict their own latents at each level via gradient descent
- data2vec analysis showing the implicit hierarchical latent prediction
Numerical results in the paper show recovery scaling as m^3 (constant in L) for latent methods versus m^(L+1) for token-level objectives across various synonym factors.
What this means for practitioners
I am not suggesting you rip out your NTP pretraining head tomorrow. The RHM is a model organism, not a web crawl.
But the direction matters for three decisions:
1. Pretraining objective choice
If your domain has clear compositional structure (code, structured documents, molecular graphs, multi-step reasoning traces), latent-prediction auxiliaries may buy sample efficiency NTP alone cannot.
Recent related work like Hierarchical Latent Prediction (HiLP) for language models adds multi-scale self-predictive learning into transformer pretraining. Inference still uses standard NTP. The hierarchy is training signal only.
2. Data budget planning
Teams often assume more tokens fixes everything. This paper says the objective may be the bottleneck when hierarchy depth is high. A smaller, structured dataset with latent prediction might outperform a larger token-only run.
3. Stacking JEPA scales
If data2vec already does hierarchical latent prediction implicitly, adding explicit H-JEPA layers may not help much. That saves architecture complexity and tuning surface.
Limits and honest caveats
- Results are on synthetic hierarchical data, not production corpora
- Constant sample complexity has logarithmic factors and depends on synonym factor m
- Transfer to billion-parameter LLM pretraining is unproven at scale
- NTP still wins at inference simplicity; latent methods are training-time auxiliaries in most designs
The paper's practical implication paragraph notes that empirical neural scaling laws in language may be governed by power-law decays of token correlations with context length. Latent prediction targets a different correlation structure entirely.
Connection to the bigger training debate
This lands in a busy June 2026 research window:
- Smaller, smarter training beats brute force (the AlphaSignal digest theme)
- Specialized LLM training APIs make custom objectives cheaper to experiment with
- Open-weight models let you modify pretraining, not just fine-tune
If you are spending six figures on pretraining tokens, understanding which objective extracts hierarchy efficiently is worth a literature review before the next run.
Training custom models or evaluating whether your data budget matches your objective? Get in touch for a focused review of your stack and training choices.