OpenAI says perplexity should be a shared metric for AI progress
OpenAI proposes perplexity as a measure of AI progress and explains why differences in evaluation datasets make scores hard to compare across labs.

- OpenAI published a research post, 'Perplexity and Intelligence', on 22 November 2026.
- The post argues that tracking progress in better compressors is key to understanding AI advances.
- OpenAI says perplexity could serve as a shared benchmark for general intelligence, while noting comparison across labs remains difficult.
- The company says smaller gains in perplexity may correspond to larger intelligence gains as models approach the low-loss regime.
OpenAI wants the industry to watch perplexity more closely
OpenAI has published a research post, “Perplexity and Intelligence,” arguing that progress in AI is best understood by tracking how well systems learn to compress large corpora of data. The post says that, rather than building intelligence from scratch, modern AI largely reflects the intelligence extracted from training a compressor or predictor on very large datasets. OpenAI says that following the pace of improvement in these compressors is key to understanding the arc of AI advances. documentation
The company also says perplexity has remained a durable research metric and could help standardise how the field talks about AI progress. In the post, OpenAI says perplexity “can play an important role” in standardising that discussion and that it hopes for more transparency across the industry. documentation
Why it matters
For developers and IT teams, this is less about a new product feature and more about how to interpret model progress. OpenAI is arguing that one metric may capture more than benchmark chasing: as models improve, lower perplexity may signal increasingly capable systems even if the visible gains look small. That matters when teams are comparing model generations, deciding when to re-test a system, or trying to explain capability shifts to non-specialists. documentation
The post goes further and suggests perplexity could become a shared benchmark for general intelligence. OpenAI says the main source of surprising new capabilities is progress on compression, and that a measure rooted in perplexity could provide a common benchmark regardless of how automated the development process becomes. The company also says that as models approach the low-loss regime, apparent progress on perplexity may slow even while the intelligence gains become larger. documentation
What OpenAI says is still hard
OpenAI does not present perplexity as a solved, universal yardstick. The post says comparison across labs is difficult because different developers use different evaluation datasets and train on different data mixtures. It also says OpenAI is reporting numbers on the main dataset it has been tracking. documentation
That limitation matters. If teams want to use perplexity as a comparative signal, they still need to know which dataset, training mixture and evaluation setup sits behind the number. Without that context, a perplexity figure can be easy to quote and hard to compare.
Comparison of the post’s claims
| Claim | What OpenAI says | Practical implication |
|---|---|---|
| Compression is central to AI progress | Track the pace of progress on training better compressors | Model progress should be read through data efficiency, not just headline features |
| Perplexity is a useful shared metric | It can standardise how the industry talks about AI progress | Teams may use it as one signal among others when evaluating models |
| Comparison is hard across labs | Different datasets and data mixtures make comparison difficult | Perplexity needs dataset context to be meaningful |
| Lower perplexity may hide bigger gains | Smaller perplexity gains may correspond to larger intelligence leaps near the low-loss regime | Apparent slowdowns may not mean capability gains have stalled |
What to watch next
OpenAI’s post is a framing document, not a product launch. The immediate question is whether other labs adopt the same language, the same datasets, or a compatible reporting convention. For now, the main takeaway is simple: OpenAI wants perplexity treated less like a niche research statistic and more like a shared way to discuss AI progress. documentation
Compression as the core lens
OpenAI says tracking better compressors is key to understanding AI advances.
Perplexity as a shared benchmark
The post says perplexity could help standardise how the industry talks about AI progress.
Cross-lab comparison remains difficult
Different datasets and data mixtures make direct comparison hard.
Small perplexity gains, larger intelligence gains
OpenAI says that near the low-loss regime, smaller gains in perplexity may correspond to larger jumps in intelligence.
