agitracker.io

What counts as AGI? Five definitions, compared

The agitracker.io team — Tue, 15 Sep 2026 00:00:00 GMT

Ask ten researchers when artificial general intelligence will arrive and you will get ten dates. Ask them what AGI is and the spread gets wider. Before we can track progress toward anything, we need to be clear about what the finish line looks like.

Here are five definitions that shape the current debate, what each one asks you to measure, and where each one breaks down.

1. The economic definition

OpenAI's 2018 charter describes AGI as "highly autonomous systems that outperform humans at most economically valuable work."

What you would measure: the share of paid tasks a system can do end to end, at human quality, without supervision.

Strength: it is concrete and consequential. If it is met, the world changes whether or not philosophers agree the system "understands" anything.

Weakness: "most economically valuable work" moves. Economies reorganise around new tools, and the work that remains valuable is often the work machines can't yet do.

2. The universal-intelligence definition

In 2007, Shane Legg and Marcus Hutter surveyed dozens of definitions of intelligence and distilled them into one: intelligence measures an agent's ability to achieve goals in a wide range of environments.

What you would measure: performance across a very broad distribution of environments, weighted toward the simple ones.

Strength: it is mathematically precise and doesn't privilege human tasks.

Weakness: the formal version is uncomputable, so in practice you still have to choose which environments to test, and the choice brings back all the bias the definition was meant to remove.

3. The skill-acquisition definition

François Chollet's 2019 paper On the Measure of Intelligence argues that skill is not intelligence. A chess engine is skilled; it isn't intelligent. Intelligence is how efficiently a system turns limited experience and prior knowledge into new skills.

What you would measure: performance on genuinely novel tasks, given very few examples. Chollet's Abstraction and Reasoning Corpus (ARC) was built for exactly this.

Strength: it targets the thing that makes humans general: learning quickly in unfamiliar situations.

Weakness: novelty is hard to guarantee. Once a benchmark is public, its "novel" tasks start leaking into training data.

4. The levels definition

A 2023 Google DeepMind paper, Levels of AGI, proposes a grid rather than a single threshold. One axis is performance (from "emerging" through "competent", "expert" and "virtuoso" to "superhuman"). The other is generality: narrow systems versus general ones.

Level	Performance, relative to skilled adults
Emerging	Equal to or somewhat better than an unskilled human
Competent	At least the 50th percentile
Expert	At least the 90th percentile
Virtuoso	At least the 99th percentile
Superhuman	Outperforms all humans

Strength: it replaces a yes/no argument with a map, so you can say "competent in these domains, emerging in those."

Weakness: placing a system on the grid still requires agreeing on which tasks count and who the "skilled adults" are.

5. The practical-tests definition

Some of the oldest definitions are simply tests. Alan Turing's 1950 imitation game asks whether a machine's conversation can be told apart from a person's. Later proposals are more physical: Steve Wozniak's "coffee test" asks a robot to walk into an unfamiliar home and make a cup of coffee. Nils Nilsson's "employment test" asks whether a machine can do the jobs people are paid to do.

Strength: anyone can understand the pass condition.

Weakness: each test captures one slice of general ability. Conversation has proved far easier to imitate than making coffee in a stranger's kitchen.

How we use these on agitracker.io

We don't pick a winner. When we report on progress, we say which definition a result speaks to:

A new score on a hard exam speaks to levels (performance in a domain).
A result on fresh, few-shot puzzles speaks to skill acquisition.
A system completing long, paid, real-world tasks speaks to the economic definition.

A claim that "AGI is here" or "AGI is decades away" is only as meaningful as the definition behind it. The first question to ask is always: by which measure?

The rest of this site is about those measures: how they work, where they fail, and what they currently show.

How to read an AI benchmark without getting fooled

The agitracker.io team — Fri, 28 Aug 2026 00:00:00 GMT

Every model launch comes with a table of benchmark scores, usually with the new model's numbers in bold. Those tables are useful, but only if you know what to look for. Here is the checklist we run through before we report a result.

1. How big is the test?

A benchmark score is an estimate, and small tests give noisy estimates. The standard error of an accuracy score is roughly:

import math

def standard_error(accuracy: float, n_questions: int) -> float:
    return math.sqrt(accuracy * (1 - accuracy) / n_questions)

# A model scoring 80% on a 500-question test
se = standard_error(0.80, 500)       # ~0.018
print(f"95% interval: ±{1.96 * se:.1%}")  # ±3.5%

On a 500-question test, a model at 80% has a 95% confidence interval of about ±3.5 points. A two-point lead on that test is not a meaningful difference. Look for results that clear the noise, or that repeat across several independent benchmarks.

2. Could the answers be in the training data?

Large models are trained on large slices of the public internet. If a benchmark's questions (or discussions of them) are online, a model may have seen them. This is called contamination, and it inflates scores without improving the underlying skill.

Signs to look for:

The benchmark has been public for years.
Performance drops sharply on freshly written questions of the same kind.
The lab doesn't say how it checked for overlap.

Benchmarks that keep a private held-out set, or that are refreshed regularly, are harder to contaminate.

3. Is the benchmark already saturated?

A benchmark is saturated when top models score close to its ceiling. At that point, differences between models mostly reflect noise, mislabelled questions and formatting quirks.

Sign	What it means
Top scores within a few points of 100%	Little headroom left to measure
Score gains of under 1 point per release	Differences are likely noise
Known label errors in the dataset	The true ceiling is below 100%

When a benchmark saturates, the field moves to a harder one. We cover that cycle in detail in why benchmarks saturate.

4. What exactly was measured?

Two scores on the "same" benchmark can come from very different setups:

Prompting. Zero-shot, few-shot, or with a long chain of reasoning?
Attempts. One answer, or best of many (often written pass@k)?
Tools. Did the model have a code interpreter, web search, or other help?
Compute. How long was the model allowed to think per question?

A score without these details can't be compared with another score. Good reports put the evaluation setup right next to the number.

5. Does the benchmark measure what it claims to?

A test of "reasoning" might mostly reward pattern matching on familiar formats. A test of "coding" might only check whether short functions pass unit tests, not whether the code is maintainable. Always ask: if a model aced this, what could it actually do?

When a measure becomes a target, it ceases to be a good measure. (Goodhart's law, paraphrased by Marilyn Strathern)

The short version

Before trusting a headline number, check:

[x] The test is large enough for the gap to matter
[x] Contamination has been addressed
[x] The benchmark isn't saturated
[x] The evaluation setup is disclosed
[x] The benchmark measures the skill in the headline

If a result passes all five, it is worth tracking. If it fails two or more, treat it as a marketing number.

Compute, data, algorithms: the three levers behind AI progress

The agitracker.io team — Fri, 10 Jul 2026 00:00:00 GMT

Most of the progress in AI over the past fifteen years comes from pulling three levers: more compute, more data and better algorithms. When you read that a new model is better, the useful question is which lever moved?

Lever 1: Compute

Training compute is the total number of calculations used to train a model, usually counted in floating-point operations (FLOP).

Epoch AI, a research group that tracks these trends, has estimated that training compute for notable models grew by roughly four to five times per year during the 2010s and early 2020s. That is far faster than chips alone improved. Most of the growth came from spending: bigger clusters, longer runs and larger budgets.

What could slow it down:

Cost. Each order of magnitude of compute costs roughly an order of magnitude more money.
Power. The largest training runs now need dedicated power planning.
Chips. Supply of leading-edge accelerators is limited.

Lever 2: Data

Models learn from examples. In 2020, researchers at OpenAI (Kaplan et al.) described scaling laws: smooth, predictable relationships between model size, data, compute and loss. In 2022, DeepMind's Chinchilla paper (Hoffmann et al.) refined them and showed that many large models had been undertrained. For a fixed compute budget, it was better to use a smaller model and train it on more tokens, around 20 tokens per parameter.

That finding made data a first-class constraint. The open questions now are:

How much high-quality human-written text is left to train on?
How well does synthetic data, generated by models themselves, substitute for it?
How much can be gained from other sources: code, video, and interaction with real environments?

Lever 3: Algorithms

Algorithmic progress means getting the same capability from less compute. Better architectures, training recipes and data filtering all count.

Epoch AI's analysis of language models estimated that the compute needed to reach a given level of performance has halved roughly every eight months, with wide uncertainty. In other words, even if hardware spending froze, efficiency gains alone would keep pushing capability forward, just more slowly.

A newer part of this lever is inference-time compute: letting a model think longer, try multiple approaches or check its own work before answering. It trades cost per answer for quality, and it has changed which benchmarks look hard.

How the levers combine

Lever	Measured in	What limits it
Compute	Training FLOP	Money, power, chip supply
Data	Tokens, quality	Supply of useful data
Algorithms	Efficiency gains	Research breakthroughs

The levers multiply rather than add. A model trained on ten times the compute, with a recipe that is twice as efficient, behaves as though it had twenty times the compute of its predecessor.

Why this matters for AGI timelines

Rich Sutton's 2019 essay The Bitter Lesson argued that, over decades, general methods that make use of more computation beat methods built on human cleverness. If that keeps holding, AGI timelines are mostly a question of how long the three levers keep moving at their current pace.

If any one of them stalls (because compute gets too expensive, data runs thin, or algorithmic gains slow), timelines stretch. We track all three for exactly that reason.

Why AI benchmarks saturate, and what replaces them

The agitracker.io team — Sat, 02 May 2026 00:00:00 GMT

Every AI benchmark has a life cycle. It is released as a hard problem, models climb it for a few years, and then it stops telling us anything. The interval between release and saturation has been getting shorter.

The cycle

Release. A new test is published, usually with a large gap between human and model performance.
Climb. New models improve steadily. The benchmark becomes a headline number in launch announcements.
Saturation. Top models approach the ceiling. Remaining errors are mostly ambiguous or mislabelled questions.
Replacement. Researchers publish a harder benchmark and the cycle repeats.

A short history

GLUE (2018) bundled several language-understanding tasks into one score. Models passed its human baseline so quickly that its creators released SuperGLUE in 2019. SuperGLUE's human baseline was passed in early 2021, less than two years later.

MMLU (2020) tested knowledge across 57 subjects, from law to physics. It became the standard knowledge benchmark for large language models, then followed the same arc toward its ceiling.

GSM8K (2021) covered grade-school maths word problems. It went from a real challenge to a near-solved test as step-by-step reasoning improved.

The replacements got harder on purpose:

GPQA (2023) used graduate-level science questions written to be "Google-proof": hard to answer even with web search.
SWE-bench (2023) asked models to resolve real issues from open-source GitHub repositories.
FrontierMath (2024) collected unpublished, research-level maths problems.
Humanity's Last Exam (2025) gathered expert-written questions at the frontier of many fields.

Why saturation keeps getting faster

Models improve on the underlying skill. This is the good reason, and it is real.

Contamination. The longer a benchmark is public, the more likely its questions end up in training data.

Targeting. Once a benchmark is a headline number, labs have an incentive to tune for it. That doesn't have to be dishonest to distort the result.

Fixed formats. Multiple-choice questions reward good guessing and elimination strategies, not only knowledge.

What the next generation looks like

The newest evaluations try to resist the cycle in a few ways:

Private test sets that are never published, so they can't leak into training data.
Living benchmarks that add fresh questions on a schedule.
Long, open-ended tasks, such as multi-hour software or research projects, that are hard to game with pattern matching.
Measuring time horizons: how long a task (measured by how long it takes a skilled human) a model can reliably complete.

A saturated benchmark isn't a failed benchmark. It is a milestone: proof that a capability once considered hard is now routine.

How we report benchmark results

On agitracker.io we flag when a benchmark is near saturation and report its successor alongside it. A gain on a saturated test is noted, not headlined. For a checklist you can use yourself, see how to read an AI benchmark.

A field guide to AGI forecasts

The agitracker.io team — Fri, 12 Dec 2025 00:00:00 GMT

"When will we get AGI?" has no single answer, but it does have several methods for producing one. Each method has its own biases. Knowing which method produced a date tells you how much weight to give it.

Expert surveys

Researchers are asked directly when they expect certain capabilities.

The largest example is the survey run by AI Impacts. Its 2023 edition collected answers from 2,778 researchers who had published at top AI venues. Asked when unaided machines would outperform humans at every possible task, the aggregate forecast gave a 50% chance by 2047. That was 13 years earlier than the same question produced in the previous year's survey.

Good at: capturing what the people building the field believe.

Watch out for: framing effects. Small changes in wording ("every task" versus "every occupation") have produced answers decades apart in the same survey.

Forecasting platforms and prediction markets

Platforms such as Metaculus let many forecasters make predictions, then score them when questions resolve. Forecasters with good track records carry more weight in the aggregate.

Good at: updating quickly when news arrives, and rewarding calibration.

Watch out for: resolution criteria. An AGI question on a forecasting site is only as meaningful as its definition of AGI, which is often a specific bundle of tests. Read the fine print before quoting a date.

Trend extrapolation

Take a measurable trend (training compute, benchmark scores, the length of tasks models can complete) and project it forward until it crosses a threshold.

Good at: being explicit. Every assumption is visible and can be argued with.

Watch out for: the assumption that trends continue. Exponential trends in the real world eventually bend, and the forecast depends heavily on when.

Compute-anchored models

These estimate how much computation a human-level system might need (sometimes anchored to estimates of the brain's computation) and then forecast when that much compute becomes affordable.

Good at: connecting timelines to physical and economic constraints.

Watch out for: very wide uncertainty in the anchor itself. Estimates of the relevant compute span many orders of magnitude.

Comparing the methods

Method	Updates quickly	Transparent assumptions	Main risk
Expert surveys	No	No	Framing effects
Forecasting platforms	Yes	Partly	Narrow resolution criteria
Trend extrapolation	Yes	Yes	Trends bending
Compute anchors	No	Yes	Uncertain anchors

How to read any AGI date

When you see a forecast, ask three questions:

Which definition of AGI does it use? (See what counts as AGI.)
Is it a median or a range? A single year hides how uncertain the forecast is.
When was it made? Forecasts have shifted considerably as capabilities have improved, so an old number may no longer reflect what the same people believe.

We keep these questions next to every forecast we report.