What is knowledge distillation

Knowledge distillation is a model compression technique where a small, efficient student model learns to replicate the behavior of a larger, more complex teacher model. The core objective is achieving comparable performance with reduced size and computational cost. The teacher transfers its knowledge to the student primarily through its soft targets—output probabilities softened using a higher temperature parameter, which reveal richer inter-class relationships than hard labels. Key factors include selecting compatible architectures, tuning the distillation temperature, and balancing the loss between mimicking the teacher's soft labels and standard training loss.

This technique enables deploying powerful deep learning models on devices with limited resources, such as mobile phones or embedded systems. Its primary business value lies in significantly reducing model size and inference latency while maintaining high accuracy, leading to lower operational costs and broader application possibilities. It is crucial for efficient scaling and real-time AI applications.