Compute, data, algorithms: the three levers behind AI progress
AI capability grows when you add compute, add data, or use them more efficiently. Here is how each lever works, how fast it has been moving, and what could slow it down.
Most of the progress in AI over the past fifteen years comes from pulling three levers: more compute, more data and better algorithms. When you read that a new model is better, the useful question is which lever moved?
Lever 1: Compute
Training compute is the total number of calculations used to train a model, usually counted in floating-point operations (FLOP).
Epoch AI, a research group that tracks these trends, has estimated that training compute for notable models grew by roughly four to five times per year during the 2010s and early 2020s. That is far faster than chips alone improved. Most of the growth came from spending: bigger clusters, longer runs and larger budgets.
What could slow it down:
- Cost. Each order of magnitude of compute costs roughly an order of magnitude more money.
- Power. The largest training runs now need dedicated power planning.
- Chips. Supply of leading-edge accelerators is limited.
Lever 2: Data
Models learn from examples. In 2020, researchers at OpenAI (Kaplan et al.) described scaling laws: smooth, predictable relationships between model size, data, compute and loss. In 2022, DeepMind’s Chinchilla paper (Hoffmann et al.) refined them and showed that many large models had been undertrained. For a fixed compute budget, it was better to use a smaller model and train it on more tokens, around 20 tokens per parameter.
That finding made data a first-class constraint. The open questions now are:
- How much high-quality human-written text is left to train on?
- How well does synthetic data, generated by models themselves, substitute for it?
- How much can be gained from other sources: code, video, and interaction with real environments?
Lever 3: Algorithms
Algorithmic progress means getting the same capability from less compute. Better architectures, training recipes and data filtering all count.
Epoch AI’s analysis of language models estimated that the compute needed to reach a given level of performance has halved roughly every eight months, with wide uncertainty. In other words, even if hardware spending froze, efficiency gains alone would keep pushing capability forward, just more slowly.
A newer part of this lever is inference-time compute: letting a model think longer, try multiple approaches or check its own work before answering. It trades cost per answer for quality, and it has changed which benchmarks look hard.
How the levers combine
| Lever | Measured in | What limits it |
|---|---|---|
| Compute | Training FLOP | Money, power, chip supply |
| Data | Tokens, quality | Supply of useful data |
| Algorithms | Efficiency gains | Research breakthroughs |
The levers multiply rather than add. A model trained on ten times the compute, with a recipe that is twice as efficient, behaves as though it had twenty times the compute of its predecessor.
Why this matters for AGI timelines
Rich Sutton’s 2019 essay The Bitter Lesson argued that, over decades, general methods that make use of more computation beat methods built on human cleverness. If that keeps holding, AGI timelines are mostly a question of how long the three levers keep moving at their current pace.
If any one of them stalls (because compute gets too expensive, data runs thin, or algorithmic gains slow), timelines stretch. We track all three for exactly that reason.
