Will knowledge distillation affect model accuracy?

Knowledge distillation typically introduces a small accuracy degradation for the student model compared to the original teacher model. However, it can potentially match or even slightly exceed teacher accuracy under specific conditions, such as when the teacher's predictions act as a form of regularization.

This accuracy impact depends critically on the relative capabilities of the teacher and student models, the distillation objective (especially the use of soft targets capturing teacher probabilities), and the distillation dataset quality. Degradation is often more noticeable when drastically reducing model size (heavy compression) or using a significantly less capable student architecture. Conversely, distillation helps preserve valuable generalization cues learned by the teacher beyond hard labels.

The primary application and value lie in efficiently compressing large, accurate models for deployment. This enables using powerful models in resource-constrained environments like edge devices or high-traffic APIs, trading minimal accuracy loss for substantial gains in inference speed and reduced computational/memory requirements, making high-performance AI feasible where the original model cannot operate.