Lilian Weng Explains Why Scaling Laws Need Careful Accounting
- AI Engineering, Software, And Developer Tooling
- AI Infrastructure, Compute, Chips, And Energy
- Capital, Markets, And Business Models
- Frontier Models And Capabilities
Watch the recap
Both papers still agree on the same underlying principle, but they disagree on where the optimal size-vs-token tradeoff lies.
Scaling-law papers often write training compute as `C ~= 6ND`: `N` is model size, `D` is training tokens, and `C` is the compute bill. If `C` is fixed, the gnarly part is choosing the split: spend more on a bigger model, or spend more on data. Weng explains that choice through two canonical scaling papers: "Scaling Laws for Neural Language Models," by OpenAI researchers led by Jared Kaplan, and "Training Compute-Optimal Large Language Models," by DeepMind researchers led by Jordan Hoffmann.
Kaplan, published in 2020, made scaling laws central for language models. It said bigger models learn more efficiently from each token, so a lab with a fixed compute budget might train a much larger model on fewer tokens and stop before the model fully converges. Chinchilla, published in 2022, tested more than 400 model-training runs and said many large models were undertrained. Its advice was simpler: if you make the model bigger, you also need many more training tokens.
both papers rely on scaling laws, but they give different advice about where to spend compute. That makes them a clean example of why the details matter. Count parameters differently, use smaller or larger test runs, round the loss number, average a fitting loss instead of summing it, or assume all tokens are equally useful, and the curve can move. Scaling laws still help labs plan expensive runs. Weng is warning that they have to be handled like measuring instruments, not slogans.
Ideas
Scaling Laws Are Planning Tools, Not Guarantees
IdeaWeng frames scaling laws as a way to forecast large training runs from smaller experiments, but only when the experiment design and assumptions behind the fit are sound.
Intro paragraphs before Early daysKaplan And Chinchilla Disagree On Allocation, Not On Scaling
IdeaThe Kaplan and Chinchilla papers both support predictable scaling, but they disagree about how to split compute between model parameters and training tokens.
Reconciling Kaplan and ChinchillaData Quality Changes What A Token Is Worth
IdeaWeng argues that modern scaling has to handle data limits because repeated or lower-quality tokens are not equivalent to fresh, high-quality data.
Scaling Laws in Data-Limited RegionSmall Fitting Choices Can Move Billion-Dollar Decisions
IdeaThe article’s strongest warning is that tiny procedural choices in fitting can become large planning errors when extrapolated from cheap runs to frontier-scale runs.
Trickiness of Fitting Scaling Laws in RealityChinchilla’s Strength Was Converging Methods
IdeaWeng emphasizes that Chinchilla was persuasive partly because three different fitting approaches pointed toward similar compute-optimal scaling.
Chinchilla Scaling Laws / methods 1-3Power Laws Have Plausible Mechanisms But No Single Settled Story
IdeaThe article gives several hypotheses for why LLM loss follows power laws, but treats them as live explanations rather than a settled theory.
Why power law?Repeated Data Can Help, But It Is Discounted
IdeaWeng’s data-limited discussion says repeated data can still improve training, but usually contributes less than new unique data and can distort learning curves.
Scaling Laws in Data-Limited Region / repeated data studiesThe Toy Simulation Is A Fitting-Risk Demo
IdeaWeng uses a toy simulation to make fitting risk tangible: rounding, tiny loss noise, and fit-region selection can change inferred scaling laws.
Toy simulation
Tags
- Frontier Models
- AI Capital Allocation
- AI Infrastructure Efficiency
- Benchmarks And Evaluation
- Compute Supply Chain