Frontier Clock

How long does the frontier stay ahead?

Many people think Chinese and open-source models are catching up to the US-led frontier. The size of the gap influences whether the US can afford an AI pause and the practicality of US-China coordination on AI safety. Frontier Clock uses Epoch AI’s benchmark data (CC BY 4.0) to track these trends and others.

Frontier curve

Frontier-equivalent date vs. release date

Gap to the frontier

Capability vs. release date

Each lab's best models

For each model, the ring (on the right) is the release date, and the dot (on the left) is the frontier-equivalent date. The line between them represents how far behind the frontier the model was when it was released. From each of 14 labs, up to three models with the best capabilities are shown, such that each model has a frontier-equivalent date within three months of that lab’s best model.

All models

Model Organization Group Release date Frontier-equivalent date Behind at release Benchmarks

* Less capable than the first frontier model in this benchmark selection, so the date is extrapolated from the trend. This happens when GPT-4 (Mar 2023) has too few scores in the selection to be placed.

Method

  1. Capability. Different models are evaluated on different benchmarks. Thus, raw scores cannot be directly compared between models. Following Epoch AI's Capabilities Index (ECI), we give each model a single capability score and each benchmark a difficulty. These values are chosen such that models score well on benchmarks easier than their capability and poorly on harder ones. This puts every model on one scale since benchmarks overlap between models. (Formally, score ≈ sigmoid(ab·(Cm − Db)), with a capability C per model and a difficulty D and slope a per benchmark.)
  2. ECI-scaled capability. Capabilities are measured on Epoch AI’s ECI scale. For all benchmarks, Claude 3.5 Sonnet’s score is pinned at 130 and GPT-5’s score is pinned at 150. Epoch pins these two models here as well. As a result, our values match Epoch’s published ECI with r = 0.997, typically within around a point. When displaying a subset of benchmarks, we rescale scores linearly such that the mean and standard deviation match those of the all-benchmarks condition so that the units are comparable. This rescaling is our own, not Epoch's official ECI methodology. Thus, our values will differ from theirs for subsets of the benchmarks.
  3. Frontier. A model is considered at the frontier if, when it was released, it was the most capable model in the world according to Epoch's data. The frontier function F(t) is built from these models either as a running max (like a staircase) or by interpolating between successive frontier models.
  4. Frontier-equivalent date. A model's frontier-equivalent date is F−1(C): the date at which the frontier first reached that model’s level of capability.
  5. Gap. For each month, we compare each group’s best model by that month with the frontier model.
  6. 90% intervals. To calculate 90% confidence intervals, we produce many resamples. Each resample makes X random draws among the benchmarks that are used, X being the number of benchmarks. In a resample, a benchmark can count multiple times or not at all; any one benchmark is left out around a third of the time. Every model has the same draws. For each resample, all the scores are refit. Then, the 5th and 95th percentile results across all resamples are taken for the intervals. The intervals ultimately indicate how much the results depend on which benchmarks are used. The benchmark scores themselves are fixed. For the all-benchmarks condition and for each individual domain, we precompute 90% intervals from 1,000 resamples. Any other selection (e.g. a combination of two domains) computes 100 resamples in your browser; as a result, its intervals are less precise (about ±0.3 months). For all benchmarks, Claude 3.5 Sonnet and GPT-5 are pinned to 130 and 150 in every refit. Therefore, they have no capability interval, and models close to them in capabilities have small intervals. For each benchmark subset, refits are rescaled to have the same mean and standard deviation as the all-benchmarks condition. This hides uncertainty about how spread apart the scores are. Thus, benchmark subset capability intervals look narrower than they actually are. Dates do not depend on the scale and are not affected.
  7. Coverage. GPT-4 (Mar 2023) is the first frontier model displayed. Models weaker than GPT-4 are left out since they cannot be dated. Benchmarks that only scored these models are also left out. Frontier Clock incorporates 180 models and 47 benchmarks, totaling 2,237 individual benchmark scores.

Caveats

Related work

Frontier Clock was inspired by Epoch AI's “Chinese AI models have lagged the US frontier by 7 months on average since 2023” and by Parv Mahajan's “The US-China AI gap is very uncertain” (MCNAIR). Both articles discuss the US-China AI capabilities gap. Frontier Clock further explores non-China open-weights models, individual labs, and specific domains.

Data and attribution

Raw benchmark scores are sourced from Epoch AI, “Capabilities & benchmarking” and used under CC BY 4.0. Frontier Clock refits capabilities with its own model. It excludes models that are weaker than GPT-4 (Mar 2023) and benchmarks that only scored such models. It also assigns benchmarks to domains and fills in missing metadata. Frontier Clock is independent and not affiliated with or endorsed by Epoch AI.

Frontier Clock is open source on GitHub.