Abstract: With the increasing scale and complexity of deep neural networks,model compression has become an essential technique for efficient deployment.Knowledge distillation enables a lightweight student model to approximate the performance of a large teacher model by learning its knowledge representation,while significantly reducing the number of parameters.Traditional distillation methods commonly apply temperature scaling to smooth the output distribution of the teacher model.However,this approach introduces empirical hyperparameter tuning and merely compresses the logits uniformly,making it difficult to preserve deep-level semantic information within the original distribution.Moreover,the discriminator in existing adversarial learning me-thods only roughly pulls the teacher and student logits closer,lacking an understanding of the overall class distribution,which can easily lead to overfitting.To address the aforementioned issues,firstly,this paper proposes a temperature-free normalization mechanism by applying z-score standardization to align the scale of logits between the teacher and student models.This approach reduces the need for manual parameter tuning and improves the stability of the alignment process.Secondly,it introduces a mutual information-enhanced adversarial distillation framework,where a mutual information term is incorporated into the discriminator loss.This design enhances the discriminator’s ability to perceive class boundaries by encouraging intra-class compactness and inter-class separability in the logits,thereby improving the discriminative capacity and classification accuracy of the student model while effectively mitigating overfitting.Experiments conducted on the CIFAR-10 and CIFAR-100 benchmark datasets demonstrate that the proposed method avoids the complexity of temperature parameter tuning and achieves approximately a 2% improvement in accuracy compared to mainstream distillation methods.

Key words: Knowledge distillation, Discriminator, Mutual information, Normalization, Temperature