融合动态稀疏性与异构知识蒸馏的Top-k推荐算法

doi:10.11896/jsjkx.250700121

摘要/Abstract

摘要： 当前的推荐算法主要侧重于利用深度学习技术提高推荐精度,进而为用户推荐其感兴趣的内容集合。然而,这类方法得到的推荐结果往往计算开销大,模型冗余度高,不适合资源受限场景。针对上述问题,提出一种融合动态稀疏性与异构知识蒸馏的协同优化框架(DySparseHKD),构建轻量化的推荐模型,在减少参数量的同时保留关键特征。提出基于交互冗余的动态稀疏率分配方法,捕获更高效的参数配置,利用教师模型训练轨迹实现渐进式知识传递,缓解异构模型间的知识差异,根据学生模型当前的学习状态动态调整知识迁移粒度,提高迁移效率;最后,通过联合优化目标实现模型复杂度与推荐性能的深度解耦。在3个真实的数据集上进行相关的实验,结果表明所提模型在具有更低复杂度的同时实现了模型效率与推荐效果的有机融合。

关键词: 动态稀疏技术, 异构知识蒸馏, 知识迁移, 协同过滤, 轻量化推荐模型

Abstract: Current recommendation algorithms mainly focus on using deep learning techniques to enhance recommendation accuracy,thereby providing users with a collection of content they are interested in.However,the recommendation results obtained by such methods often have high computational costs and model redundancy,making them unsuitable for resource-constrained scenarios.To address these issues,a collaborative optimization framework(DySparseHKD) that integrates dynamic sparsity and he-terogeneous knowledge distillation is proposed.This framework builds a lightweight recommendation model that reduces the number of parameters while retaining key features.A dynamic sparsity rate allocation method based on interaction redundancy is proposed to capture more efficient parameter configurations.The training trajectory of the teacher model is utilized to achieve progressive knowledge transfer,alleviating the knowledge gap between heterogeneous models.The knowledge transfer granularity is dynamically adjusted according to the current learning state of the student model to improve transfer efficiency.Finally,the deep decoupling of model complexity and recommendation performance is achieved through joint optimization objectives.Experiments on three real datasets show that the proposed model achieves an organic integration of model efficiency and recommendation effect while maintaining lower complexity.

Key words: Dynamic sparse technology, Heterogeneous knowledge distillation, Knowledge transfer, Collaborative filtering, Lightweight recommendation model

中图分类号:

TP301

付诗棋, 朱金侠, 许琪晨, 杜泽宇. 融合动态稀疏性与异构知识蒸馏的Top-k推荐算法[J]. 计算机科学, 2026, 53(6A): 250700121-9. https://doi.org/10.11896/jsjkx.250700121

FU Shiqi, ZHU Jinxia, XU Qichen, DU Zeyu. Dynamic Sparsity and Heterogeneous Knowledge Distillation for Top-k Recommendation[J]. Computer Science, 2026, 53(6A): 250700121-9. https://doi.org/10.11896/jsjkx.250700121

参考文献

[1] JIN Y,LI M K,LU Y,et al.Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation[C]//CVPR 2023:IEEE Conference on Computer Vision and Pattern Recognition.Vancouver,BC Canada:IEEE,2023:23695-23704.
[2] ZHANG B C,SUN C G,TAN J C,et al.SHARK:A Lightweight Model Compression Approach for Large-scale Recommender Systems[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management.New York,NY,United States:Association for Computing Machinery,2023:4930-4937.
[3] ADNAN M,EBRAHIMZADEH M Y,MAHAIAN D,et al.Nair:Heterogeneous Acceleration Pipeline for Recommendation System Training[C]//International Symposium on Computer Architecture(ISCA 2024).Buenos Aires,Argentina:IEEE,2024:1063-1079.
[4] ISINKAYE F O.Matrix Factorization in Recommender Sys-tems:Algorithms,Applications,and Peculiar Challenges[J].IETE Journal of Research,2021,69(9):6087-6100.
[5] GAO C,ZHENG Y,LI N,et al.A Survey of Graph Neural Networks for Recommender Systems:Challenges,Methods,and Directions[J].Trans.Recomm.Syst.,2023,1(1):1-51.
[6] HUANG W Z,ZHANG Y X,ZHENG X W,et al.DynamicLow-Rank Sparse Adaptation for Large Language Models[J].arXiv:2502.14816,2025.
[7] KANG S K,HWANG J Y,KWEON W B,et al.DERRD:AKnowledge Distillation Framework for Recommender System[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management.New York,NY,United States:Association for Computing Machinery,2020:605-614.
[8] KANG S K,HWANG J Y,KWEON W B,et al.ItemsideRanking Regularized Distillation for Recommender System[J].Information Sciences,2021,580(2021):15-34.
[9] YU S,HE X L,CHEN K,et al.HKDSME:HeterogeneousKnowledge Distillation for Semi-supervised Singing Melody Extraction Using Harmonic Supervision[C]//Proceedings of the 32nd ACM International Conference on Multimedia.New York,NY,United States:Association for Computing Machinery,2024:545-553.
[10] KANG S K,KWEON W D,DONGHA LEE D,et al.Distillation from Heterogeneous Models for Top-k Recommendation[C]//Proceedings of the ACM Web Conference 2023.New York,NY,United States:Association for Computing Machinery,2023:801-811.
[11] HAN S,MAO H Z,DALLY W J.Deep Compression:Compres-sing Deep Neural Networks with Pruning,Trained Quantization and Huffman Coding[C]//Proceedings of the International Conference on Learning Representations 2016.Barcelona,Spain:ICLR,2016:1131-1135.
[12] LIU Z,LI J G,SHEN Z Q,et al.Learning Efficient Convolutional Networks through Network Slimming[C]//ICCV 2017:International Conference on Computer Vision.Venice,Italy:IEEE,2017:2755-2763.
[13] MOCANU D C,MOCANU E,STONE P,et al.EvolutionaryTraining of Sparse Artificial Neural Networks:A Network Science Perspective[J].arXiv:1707.04780v1,2017.
[14] FRANKLE J,CARBIN M.The Lottery Ticket Hypothesis:Finding Sparse,Trainable Neural Networks[C]//Proceedings of the International Conference on Learning Representations.New Orleans,United States:ICLR,2019:8359-8401.
[15] HINTON G E,VINYALS O,DEAN J.Distilling the Knowledge in a Neural Network[J].arXiv:1503.02531,2015.
[16] LIU D G,CHENG P X,DONG Z H,et al.A General Knowledge Distillation Framework for Counterfactual Recommendation via Uniform Data[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.New York,NY,United States:Association for Computing Machinery,2020:831-840.
[17] HOSSAIN M I,AKHTER S H,HONG C S,et al.SingleTeacher,Multiple Perspectives:Teacher Knowledge Augmentation for Enhanced Knowledge Distillation[C]//Proceedings of the International Conference on Learning Representations.Singapore:ICLR,2025:503-524.
[18] ZHANG Y L,SU J,ZHAO H Y.Knowledge-aware and Interactive Multi-view Distillation Recommendation Algorithm[J].Computer,2024,34(3):587-634.
[19] CHANG X Q,LI Y M,LI Z C,et al.An Emotional Classification Method Based on Multi-teacher and Multi-student Knowledge Distillation[J].Journal of Chinese Information Technology,2024,38(10):127-134.
[20] YOU S,XU C,XU C,et al.Learning from Multiple Teacher Networks[C]//Proceedings of the 23rd ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining.New York,NY,United States:Association for Computing Machinery,2017:1285-1294.
[21] CHEN X,ZHANG Y F,XU H T,et al.Adversarial Distillation for Efficient Recommendation with External Knowledge[J].ACM Trans.Inf.Syst.,2019,37(1):12:1-12:28.
[22] XIA X,YIN H Z,YU J L,et al.On-Device Next-Item Recommendation with Self Supervised Knowledge Distillation[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.New York,NY,United States:Association for Computing Machinery,2022:546-555.
[23] WEI W,TANG J B,XIA L H,et al.PromptMM:Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tu-ning[C]//Proceedings of the ACM Web Conference 2024.New York,NY,United States:Association for Computing Machinery,2024:3217-3228.
[24] ZHU J X,MENG X F,XING CH Z,et al.Collaborative filtering recommendation approach fused with graph convolutional attention mechanism[J].CAAI Transactions on Intelligent Systems,2023,18(6):1295-1304.
[25] HE F Z,SHI J P.A Diversity Recommendation Method Under the Constraint of Non-uniform Division Matrix[J].Computer Science and Exploration,2019,13(2):226-238.
[26] KAUR K,CHADHA M,GUPTA V,et al.Efficient and Responsible Adaptation of Large Language Models for Robust and Equitable Top-k Recommendations[J].arXiv:2501.04762,2025.
[27] CHEN Z F,GAN W S,WU J Y,et al.Data Scarcity in Recommendation Systems:A Survey[J].ACM Transactions on Re-commender Systems,2025,24(9):3:1-31.
[28] LEE Y J,KIM K E.Dual Correction Strategy for RankingDistillation in Top-N Recommender System[C]//Proceedings of the 30th ACM International Conference on Information & Know-ledge Management.New York,NY,United States:Association for Computing Machinery,2021:3186-3190.
[29] KEMERTAS M,PISHDAD L,DERPANIS K G,et al.RankMI:A Mutual Information Maximizing Ranking Loss[C]//CVPR 2020:IEEE Conference on Computer Vision and Pattern Recognition.Seattle,WA,USA:IEEE,2020:14350-14359.
[30] KANG S K,HWANG J Y,KWEON W B,et al.DERRD:AKnowledge Distillation Framework for Recommender System[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management.New York,NY,United States:Association for Computing Machinery,2020:605-614.
[31] ZENG H S,ZAMANI H,VINAY V.Curriculum Learning forDense Retrieval Distillation[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.New York,NY,United States:Asso-ciation for Computing Machinery,2022:1979-1983.
[32] KWEON W,KANG S K,YU H.Bidirectional Distillation forTop-K Recommender System[C]//Proceedings of the Web Conference 2021.New York,NY,United States:Association for Computing Machinery,2021:3861-3871.
[33] LEE Y J,KIM K E.Dual Correction Strategy for Ranking Distillation in Top-N Recommender System[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.New York,NY,United States:Association for Computing Machinery,2021:3186-3190.
[34] RENDLE S,FREUDENTHALER C H,GANTNER Z,et al.BPR:Bayesian personalized ranking from implicit feedback[C]//Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence.Arlington,Virginia,United States:AUAI Press,2009:452-461.
[35] HSIEH C K,YANG L Q,CUI Y,et al.Collaborative metriclearning[C]//Proceedings of the 26th International Conference on World Wide Web.Republic and Canton of Geneva,Switzerland:International World Wide Web Conferences Steering Committee,2017:193-201.
[36] HE X N,LIAO L Z,ZHANG H W,et al.Neural collaborative filtering[C]//Proceedings of the 26th International Conference on World Wide Web.Republic and Canton of Geneva,Switzerland:International World Wide Web Conferences Steering Committee,2017:173-182.
[37] HE X N,DENG K,WANG X,et al.LightGCN:Simplifying andPowering Graph Convolution Network for Recommendation[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.New York,NY,United States:Association for Computing Machinery,2020:639-648.
[38] LIANG D W,KRISHNAN R G,HOFFMAN M D,et al.Variational Autoencoders for Collaborative Filtering[C]//Procee-dings of the 2018 World Wide Web Conference.Republic and Canton of Geneva,Switzerland:International World Wide Web Conferences Steering Committee,2018:689-698.
[39] SEDHAIN S,MENON A K,SANNER S,et al.Autorec:Autoencoders meet collaborative filtering[C]//Proceedings of the 24th International Conference on World Wide Web.New York,NY,United States:Association for Computing Machinery,2015:111-112.
[40] WANG X,LI Y,ZHANG H.DSL:Dynamic Sparse Learning for Efficient and Accurate Recommendation.[C]//Proceedings of the 17th ACM International Conference on Web Search and Data Mining.2024:123-132.
[41] CHEN J,LIU S,WANG Z.LightGNN:Sparse Scoring andMulti-Level Knowledge Distillation for Efficient Graph Neural Recommendation.[C]//Proceedings of the 18th ACM International Conference on Web Search and Data Mining.2025:456-465.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed