计算机科学

计算机科学 ›› 2026, Vol. 53 ›› Issue (9): 283-290.doi: 10.11896/jsjkx.251000054

• 人工智能 • 上一篇    下一篇

融合温度去除机制与互信息增强策略的对抗知识蒸馏框架

赵俊龙, 徐彬, 辛威威   

  1. 东北大学计算机科学与工程学院 沈阳 110167
  • 收稿日期:2025-10-15 修回日期:2026-02-02 出版日期:2026-09-15 发布日期:2026-09-10
  • 通讯作者: 徐彬(xubin@mail.neu.edu.cn)
  • 作者简介:(zhaojunlong0@outlook.com)

Adversarial Knowledge Distillation Framework Integrating Temperature Removal Mechanism and Mutual Information Enhancement Strategy

ZHAO Junlong, XU Bin, XIN Weiwei   

  1. School of Computer Science and Engineering,Northeastern University,Shenyang 110167,China
  • Received:2025-10-15 Revised:2026-02-02 Published:2026-09-15 Online:2026-09-10
  • About author:ZHAO Junlong,born in 2000,postgra-duate.His main research interests include large language model and know-ledge distillation.
    XU Bin,born in 1980,Ph.D,associate professor,is a member of CCF(No.21664S).His main research interests include artificial intelligence and smart education.

PDF (PC)

256

摘要： 随着深度神经网络规模与复杂度的增加,模型压缩技术已成为实现高效部署模型的重要手段。知识蒸馏通过引导结构更简单的学生模型学习教师模型的知识表示,在显著减少参数量的同时保持接近的性能。然而,传统蒸馏方法依赖温度缩放平滑教师模型的输出分布,既需要大量经验调参,又只能整体压缩logits,难以保留原始分布中的深层特征。此外,现有对抗学习中的判别器仅笼统地拉近教师与学生logits,缺乏对整体类别分布的理解,易导致过拟合。针对上述问题,首先,提出一种温度去除机制,采用z-score标准化统一教师与学生logits的尺度,减少参数调节,同时提升对齐过程的稳定性;其次,提出一种互信息增强的对抗蒸馏机制,在判别器损失中引入互信息项,以强化类别边界感知,促使同类logits聚集、异类分离,从而提升学生模型的区分能力和判别准确率,并有效抑制过拟合。在CIFAR-10,CIFAR-100基准数据集上的实验结果表明,所提方法不仅有效避免了传统温度参数调整的复杂性,还在模型准确率上较主流知识蒸馏方法提升了约2%。

关键词: 知识蒸馏, 判别器, 互信息, 标准化, 温度

Abstract: With the increasing scale and complexity of deep neural networks,model compression has become an essential technique for efficient deployment.Knowledge distillation enables a lightweight student model to approximate the performance of a large teacher model by learning its knowledge representation,while significantly reducing the number of parameters.Traditional distillation methods commonly apply temperature scaling to smooth the output distribution of the teacher model.However,this approach introduces empirical hyperparameter tuning and merely compresses the logits uniformly,making it difficult to preserve deep-level semantic information within the original distribution.Moreover,the discriminator in existing adversarial learning me-thods only roughly pulls the teacher and student logits closer,lacking an understanding of the overall class distribution,which can easily lead to overfitting.To address the aforementioned issues,firstly,this paper proposes a temperature-free normalization mechanism by applying z-score standardization to align the scale of logits between the teacher and student models.This approach reduces the need for manual parameter tuning and improves the stability of the alignment process.Secondly,it introduces a mutual information-enhanced adversarial distillation framework,where a mutual information term is incorporated into the discriminator loss.This design enhances the discriminator’s ability to perceive class boundaries by encouraging intra-class compactness and inter-class separability in the logits,thereby improving the discriminative capacity and classification accuracy of the student model while effectively mitigating overfitting.Experiments conducted on the CIFAR-10 and CIFAR-100 benchmark datasets demonstrate that the proposed method avoids the complexity of temperature parameter tuning and achieves approximately a 2% improvement in accuracy compared to mainstream distillation methods.

Key words: Knowledge distillation, Discriminator, Mutual information, Normalization, Temperature

中图分类号: 

  • TP183
[1] HINTON G,VINYALS O,DEAN J.Distilling the knowledge in a neural network[J].arXiv:1503.02531,2015.
[2] LI Z,LI X,YANG L,et al.Curriculum temperature for know-ledge distillation[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2023:1504-1512.
[3] WANG L,YOON K J.Knowledge distillation and student-teacher learning for visual intelligence:A review and new outlooks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(6):3048-3068.
[4] CHEN D,MEI J P,ZHANG Y,et al.Cross-layer distillationwith semantic calibration[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2021:7028-7036.
[5] CHEN H,WANG Y,XU C,et al.Learning student networks via feature embedding[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(1):25-35.
[6] CUI J,KINGSBURY B,RAMABHADRAN B,et al.Knowledge distillation across ensembles of multilingual models for low-resource languages[C] //2017 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2017:4825-4829.
[7] ROMERO A,BALLAS N,KAHOU S E,et al.FitNets:Hints for Thin Deep Nets[C] //ICLR.2015.
[8] ZAGORUYKO S,KOMODAKIS N.Paying More Attention toAttention:Improving the Performance of Convolutional Neural Networks via Attention Transfer[C] //International Conference on Learning Representations.2017.
[9] SUN S,REN W,LI J,et al.Logit standardization in knowledge distillation[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:15731-15740.
[10] CHAN W,KE N R,LANE I R.Transferring Knowledge from a RNN to a DNN [J].arXiv:1504.01483v1,2015.
[11] DING Q,WU S,SUN H,et al.Adaptive regularization of labels[J].arXiv:1908.05474,2019.
[12] DU S,YOU S,LI X,et al.Agree to disagree:Adaptive ensemble knowledge distillation in gradient space[J].Advances in Neural Information Processing Systems,2020,33:12345-12355.
[13] FREITAG M,AL-ONAIZAN Y,SANKARAN B.Ensembledistillation for neural machine translation[J].arXiv:1702.01802,2017.
[14] FU H,ZHOU S,YANG Q,et al.LRC-BERT:latent-representation contrastive knowledge distillation for natural language understanding[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2021:12830-12838.
[15] MIRZADEH S I,FARAJTABAR M,LI A,et al.Improvedknowledge distillation via teacher assistant[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2020:5191-5198.
[16] KIM J,PARK S U,KWAK N.Paraphrasing complex network:Network compression via factor transfer[C] //NIPS’18:Procee-dings of the 32nd International Conference on Neural Information Processing Systems.2018:2765 - 2774.
[17] LIU Y,CAO J,LI B,et al.Knowledge distillation via instance relationship graph[C] //Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2019:7096-7104.
[18] ZHANG Y,XIANG T,Hospedales T M,et al.Deep mutuallearning[C] //Proceedings of the IEEE Conference on Compu-ter Vision and Pattern Recognition.2018:4320-4328.
[19] CHEN H,WANG Y,XU C,et al.Data-free learning of student networks[C] //Proceedings of the IEEE/CVF International Conference on Computer Vision.2019:3514-3522.
[20] GUPTA S,HOFFMAN J,MALIK J.Cross modal distillationfor supervision transfer[C] //Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2016:2827-2836.
[21] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C] //NIPS’14:Proceedings of the 28th International Conference on Neural Information Processing Systems.2014:2672-2680.
[22] WANG J,YU L,ZHANG W,et al.IRGAN:A minimax game for unifying generative and discriminative information retrieval models[C] //SIGIR 2017-Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval.ACM,2017:515-524.
[23] XU Z,HSU Y C,HUANG J.Learning loss for knowledge distillation with conditional adversarial networks[J].arXiv:1709.00513,2017.
[24] WANG X,ZHANG R,SUN Y,et al.Kdgan:Knowledge distillation with generative adversarial networks[C] //Proceedings of the 32nd International Conference on Neural Information Processing Systems(NIPS’18).2018,783-794.
[25] LI C,XU T,ZHU J,et al.Triple Generative Adversarial Nets[C] //Advances in Neural Information Processing Systems.2017:4088-4098.
[26] YU L,ZHANG W,WANG J,et al.SeqGAN:Sequence Generative Adversarial Nets with Policy Gradient[C] //Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.2017:2852-2858.
[27] MADDISON C J,MNIH A,TEH Y W.The Concrete Distribution:A Continuous Relaxation of Discrete Random Variables[C] //Proceedings of the International Conference on Learning Representations(ICLR).2017.
[28] JANG E,GU S,POOLE B.Categorical Reparameterization with Gumbel-Softmax[C] //International Conference on Learning Representations.2017.
[29] SAU B B,BALASUBRAMANIAN V N.Deep model compression:Distilling knowledge from noisy teachers[J].arXiv:1610.09650,2016.
[30] SINAGA M A.On study of mutual information and its estimation methods[J].arXiv:2106.14646,2021.
[31] CZYZ· P,GRABOWSKI F,VOGT J,et al.Beyond normal:On the evaluation of mutual information estimators[J].Advances in Neural Information Processing Systems,2023,36:16957-16990.
[32] BELGHAZI M I,BARATIN A,RAJESWAR S,et al.Mine:mutual information neural estimation[J].arXiv:1801.04062,2018.
[33] JAMES W,STEIN C.Estimation with quadratic loss[C] //Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability.1961:361-379.
[34] WANG Z,SHI L,MEI Z,et al.Iterative transfer knowledge distillation and channel pruning for unsupervised cross-domain compression [C] //International Conference on Web Information Systems and Applications.Singapore:Springer,2024:3-15.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市两江新区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发