Lilian Weng Explains Why Scaling Laws Need Careful Accounting

Scaling-law papers often write training compute as `C ~= 6ND`: `N` is model size, `D` is training tokens, and `C` is the compute bill. If `C` is fixed, the gnarly part is choosing the split: spend more on a bigger model, or spend more on data. Weng explains that choice through two canonical scaling papers: "Scaling Laws for Neural Language Models," by OpenAI researchers led by Jared Kaplan, and "Training Compute-Optimal Large Language Models," by DeepMind researchers led by Jordan Hoffmann.

Kaplan, published in 2020, made scaling laws central for language models. It said bigger models learn more efficiently from each token, so a lab with a fixed compute budget might train a much larger model on fewer tokens and stop before the model fully converges. Chinchilla, published in 2022, tested more than 400 model-training runs and said many large models were undertrained. Its advice was simpler: if you make the model bigger, you also need many more training tokens.

both papers rely on scaling laws, but they give different advice about where to spend compute. That makes them a clean example of why the details matter. Count parameters differently, use smaller or larger test runs, round the loss number, average a fitting loss instead of summing it, or assume all tokens are equally useful, and the curve can move. Scaling laws still help labs plan expensive runs. Weng is warning that they have to be handled like measuring instruments, not slogans.