Can perplexity be used to compare different models?

Perplexity is a standard metric used to compare different language models, particularly in evaluating their prediction capabilities. Yes, it can be directly applied for such comparisons.

Perplexity quantifies how well a probability model predicts a sample, with lower values indicating better predictive performance and lower uncertainty. For valid comparison, models must be evaluated on the exact same test dataset and vocabulary. It is most reliable when comparing models of the same type or architecture within natural language processing tasks. However, caution is needed as perplexity primarily measures intrinsic performance (how well the model predicts its training-like data) and may not perfectly correlate with extrinsic performance in real-world tasks or user experience.

Perplexity's primary application value lies in benchmarking during model development and selection. It allows researchers and engineers to objectively rank models for text generation quality, track improvement across iterations, and inform optimization strategies. This makes it indispensable for optimizing language models like LLMs on standard corpora.