Why use perplexity to measure models

Perplexity quantifies how accurately a probabilistic model, particularly language models, predicts a sample of text. It serves as a core intrinsic evaluation metric directly reflecting a model's prediction confidence.

Perplexity is calculated based on the inverse probability the model assigns to test data, normalized by word count. A lower perplexity score indicates the model finds the test data less "surprising," signifying better predictive performance. It enables the comparison of different models or architectures trained on similar data. Perplexity is also valuable for tuning model hyperparameters without costly human evaluations, as it's computed directly from the model's output probabilities against a held-out dataset.

Measuring perplexity provides an efficient, quantitative assessment of a language model's fundamental ability to estimate word sequences. Optimizing for lower perplexity during training often correlates strongly with improved fluency and coherence in generated text. However, it primarily assesses prediction probability rather than semantic accuracy, task-specific utility, or human preference, which should be evaluated separately. It remains a vital pre-deployment checkpoint.