OpenAI says perplexity should be a shared metric for AI progress

OpenAI proposes perplexity as a measure of AI progress and explains why differences in evaluation datasets make scores hard to compare across labs.

Editorial illustration of an AI research dashboard with a falling perplexity curve and research notes about AI progress.
AI-generated editorial illustration (OpenAI)
Key takeaways
  • OpenAI published a research post, 'Perplexity and Intelligence', on 22 November 2026.
  • The post argues that tracking progress in better compressors is key to understanding AI advances.
  • OpenAI says perplexity could serve as a shared benchmark for general intelligence, while noting comparison across labs remains difficult.
  • The company says smaller gains in perplexity may correspond to larger intelligence gains as models approach the low-loss regime.

OpenAI wants the industry to watch perplexity more closely

OpenAI has published a research post, “Perplexity and Intelligence,” arguing that progress in AI is best understood by tracking how well systems learn to compress large corpora of data. The post says that, rather than building intelligence from scratch, modern AI largely reflects the intelligence extracted from training a compressor or predictor on very large datasets. OpenAI says that following the pace of improvement in these compressors is key to understanding the arc of AI advances. documentation

The company also says perplexity has remained a durable research metric and could help standardise how the field talks about AI progress. In the post, OpenAI says perplexity “can play an important role” in standardising that discussion and that it hopes for more transparency across the industry. documentation

Why it matters

For developers and IT teams, this is less about a new product feature and more about how to interpret model progress. OpenAI is arguing that one metric may capture more than benchmark chasing: as models improve, lower perplexity may signal increasingly capable systems even if the visible gains look small. That matters when teams are comparing model generations, deciding when to re-test a system, or trying to explain capability shifts to non-specialists. documentation

The post goes further and suggests perplexity could become a shared benchmark for general intelligence. OpenAI says the main source of surprising new capabilities is progress on compression, and that a measure rooted in perplexity could provide a common benchmark regardless of how automated the development process becomes. The company also says that as models approach the low-loss regime, apparent progress on perplexity may slow even while the intelligence gains become larger. documentation

What OpenAI says is still hard

OpenAI does not present perplexity as a solved, universal yardstick. The post says comparison across labs is difficult because different developers use different evaluation datasets and train on different data mixtures. It also says OpenAI is reporting numbers on the main dataset it has been tracking. documentation

That limitation matters. If teams want to use perplexity as a comparative signal, they still need to know which dataset, training mixture and evaluation setup sits behind the number. Without that context, a perplexity figure can be easy to quote and hard to compare.

Comparison of the post’s claims

Claim What OpenAI says Practical implication
Compression is central to AI progress Track the pace of progress on training better compressors Model progress should be read through data efficiency, not just headline features
Perplexity is a useful shared metric It can standardise how the industry talks about AI progress Teams may use it as one signal among others when evaluating models
Comparison is hard across labs Different datasets and data mixtures make comparison difficult Perplexity needs dataset context to be meaningful
Lower perplexity may hide bigger gains Smaller perplexity gains may correspond to larger intelligence leaps near the low-loss regime Apparent slowdowns may not mean capability gains have stalled

What to watch next

OpenAI’s post is a framing document, not a product launch. The immediate question is whether other labs adopt the same language, the same datasets, or a compatible reporting convention. For now, the main takeaway is simple: OpenAI wants perplexity treated less like a niche research statistic and more like a shared way to discuss AI progress. documentation

OpenAI’s perplexity argument at a glance · 2026-11-22
Compression as the core lens

OpenAI says tracking better compressors is key to understanding AI advances.

Perplexity as a shared benchmark

The post says perplexity could help standardise how the industry talks about AI progress.

Cross-lab comparison remains difficult

Different datasets and data mixtures make direct comparison hard.

Small perplexity gains, larger intelligence gains

OpenAI says that near the low-loss regime, smaller gains in perplexity may correspond to larger jumps in intelligence.

AI assistance · OpenAI assisted with research, drafting, verification and illustration.

Sources

Tags · OpenAI · AI research · benchmarking · perplexity · model evaluation

Related reading