基于注意力机制的可解释点击率预估模型研究

doi:10.11896/jsjkx.221000032

摘要/Abstract

摘要： 在推荐系统研发中,点击率(Click-Through Rate,CTR)预估是非常重要的工作,点击率预估精度的提升直接影响到整个推荐系统的收益,对其性能和解释性的研究有助于理解系统决策的机理,同时还能帮助优化需求和系统设计。当前点击率预估深度模型多基于线性特征交互和深度特征提取进行设计。由于深度模型的黑盒特点,该类模型在解释性方面存在局限性,并且在先前的研究中,对点击率预估模型的解释性研究非常少。因此,文中基于多头自注意力机制,对该类模型的解释性进行研究,通过多头注意力机制对特征嵌入、线性特征交互和深度部分进行增强和解释,在深度部分设计了两种模型,即注意力增强的深度神经网络和注意力叠加的深度模型,通过计算每个模块的注意力得分对其进行解释。所提方法在多个真实数据集上进行了大量实验,结果表明所提方法能够有效提升模型效果,并且模型自身带有一定的解释性。

Abstract: Click-Through Rate(CTR) prediction is critical to recommender systems.The improvement of CTR prediction can directly affect the earnings target of the recommender system.The performance and interpretation of the CTR prediction algorithm can guide developers to understand and evaluate recommender system accurately.That's also helpful for system design.Most existing approaches are based on linear feature interaction and deep feature extraction,which have poor model interpretation in the outcomes.Moreover,very few previous studies were conducted on the model interpretation of the CTR prediction.Therefore,in this paper,we propose a novel model which introduces multi-head self-attention mechanism to the embedding layer,the linear feature interaction component and the deep component,to study the model interpretation.We propose two models for the deep component.One is deep neural networks(DNN) enhanced by multi-head self-attention mechanism,the other computes high-order feature interaction by stacking multiple attention blocks.Furthermore,we calculate attention scores and interpret the prediction results for each component.We conduct extensive experiments using three real-world benchmark datasets.The results show that the proposed approach not only improves the effect of DeepFM effectively but also offers good model interpretation.

Key words: Recommender system, Click-Through Rate prediction, Multi-head self-attention mechanism, Feature interaction, Model interpretability

中图分类号:

TP391

杨斌, 梁婧, 周佳薇, 赵梦赐. 基于注意力机制的可解释点击率预估模型研究[J]. 计算机科学, 2023, 50(5): 12-20. https://doi.org/10.11896/jsjkx.221000032

YANG Bin, LIANG Jing, ZHOU Jiawei, ZHAO Mengci. Study on Interpretable Click-Through Rate Prediction Based on Attention Mechanism[J]. Computer Science, 2023, 50(5): 12-20. https://doi.org/10.11896/jsjkx.221000032

参考文献

[1]XIANG L.Recommender system practice[M].People PostPress,2012.
[2]WANG Z.Deep Learning Recommender System[M].Electronic Industry Press,2020.
[3]GUO H,TANG R,YE Y,et al.DeepFM:a factorization-machine based neural network for CTR prediction[J].arXiv:1703.04247,2017.
[4]LIU M J,ZENG G C,YUE W,et al.Review on click-through rate prediction models for display advertising[J].Computer Science,2019,46(7):38-49.
[5]CHEN T,GUESTRIN C.Xgboost:A scalable tree boosting system[C]//Proceedings of the 22ndACM Sigkdd International Conference on Knowledge Discovery and Data Mining.2016:785-794.
[6]HE X J,PAN W J,CHENG H.An advertisement click-through rate prediction model based on ensemble learning[J].Computing Engineering & Science,2019,41(12):2278-2284.
[7]RENDLE S.Factorization machines[C]//2010 IEEE Interna-tional Conference on Data Mining.IEEE,2010:995-1000.
[8]JUAN Y,ZHUANG Y,CHIN W S,et al.Field-aware factorization machines for CTR prediction[C]//Proceedings of the 10th ACM Conference on Recommender Systems.2016:43-50.
[9]HE X,PAN J,JIN O,et al.Practical lessons from predicting clicks on ads at facebook[C]//Proceedings of the Eighth International Workshop on Data Mining for Online Advertising.2014:1-9.
[10]ZHANG W,DU T,WANG J.Deep learning over multi-field ca-tegorical data[C]//European Conference on Information Retrie-val.Cham:Springer,2016:45-57.
[11]CHENG H T,KOC L,HARMSEN J,et al.Wide & deep lear-ning for recommender systems[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems.2016:7-10.
[12]WANG R,FU B,FU G,et al.Deep & cross network for ad click predictions[M]//Proceedings of the ADKDD' 17.2017:1-7.
[13]LIAN J,ZHOU X,ZHANG F,et al.xdeepfm:Combining expli-cit and implicit feature interactions for recommender systems[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.2018:1754-1763.
[14]NING T,MIAO D Z,DONG Q W,et al.Wide and deep learning for default risk prediction[J].Computer Science,2021,48(5):197-201.
[15]DING Y,LEI X,LIAO B,et al.MLRDFM:a multi-view Laplacian regularized DeepFM model for predicting miRNA-disease associations[J].Briefings in Bioinformatics,2022,23(3):bbac079.
[16]CAO B Q,XIAO Q X,ZHANG X P,et al.An API service re-commendation method via combining self-organization map-based functionality clustering and deep factorization machine-based quality prediction[J].Chinese Journal of Computers,2019,42(6):1367-1383.
[17]VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Advances in Neural Information Processing Systems,2017:6000-6010.
[18]HAN K,WANG Y,CHEN H,et al.A survey on vision transformer[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(1):87-110.
[19]XIAO J,YE H,HE X,et al.Attentional factorization machines:Learning the weight of feature interactions via attention networks[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence.2017:3119-3125.
[20]SONG W,SHI C,XIAO Z,et al.AutoInt:Automatic feature interaction learning via self-attentive neural networks[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.2019:1161-1170.
[21]HUANG T,ZHANG Z,ZHANG J.FiBiNET:combining feature importance and bilinear feature interaction for click-through rate prediction[C]//Proceedings of the 13th ACM Conference on Recommender Systems.2019:169-177.
[22]YU R,YE Y,LIU Q,et al.Xcrossnet:Feature structure-oriented learning for click-through rate prediction[C]//Pacific-Asia Conference on Knowledge Discovery and Data Mining.Cham:Springer,2021:436-447.
[23]WANG R,SHIVANNA R,CHENG D,et al.Dcn v2:Improved deep & cross network and practical lessons for web-scale lear-ning to rank systems[C]//Proceedings of the Web Conference 2021.2021:1785-1797.
[24]CHEFER H,GUR S,WOLF L.Transformer interpretability beyond attention visualization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:782-791.
[25]HAO Y,DONG L,WEI F,et al.Self-attention attribution:In-terpreting information interactions inside transformer[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(14):12963-12971.
[26]LIN Z,FENG M,SANTOS C N,et al.A structured self-attentive sentence embedding[C]//Proceedings of the International Conference on Learning Representations.2017.
[27]TU D D,SHU C C,YU H Y.Using unified probabilistic matrix factorization for contextual advertisement recommendation[J].RuanJian Xue Bao/Journal of Software,2013,24(3):454-464.
[28]HE X,CHUA T S.Neural factorization machines for sparse pre-dictive analytics[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval.2017:355-364.
[29]BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv:1409.0473,2014.
[30]DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training ofdeep bidirectional transformers for language understanding[J].arXiv:1810.04805,2018.
[31]LIANG B,LIU Q,XU J,et al.Aspect-based sentiment analysis based on multi-attention CNN[J].Journal of Computer Research and Development,2017,54(8):1724-1735.
[32]WANG W G,SHEN J B,JIA Y D.Review of visual attentiondetection[J].Ruan Jian Xue Bao/Journal of Software,2019,30(2):416-439.
[33]PARK D H,HENDRICKS L A,AKATA Z,et al.Multimodal explanations:Justifying decisions and pointing to the evidence[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8779-8788.
[34]CHOI E,BAHADORI M T,SUN J,et al.Retain:An interpretable predictive model forhealthcare using reverse time attention mechanism[C]//Advances in Neural Information Processing Systems.2016:3512-3520.
[35]LEE K,ORTEN B,DASDAN A,et al.Estimating conversionrate in display advertising from past erformance data[C]//Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.2012:768-776.
[36]REN K,ZHANG W,RONG Y,et al.User response learning for directly optimizing campaign performance in display advertising[C]//Proceedings of the 25th ACM International on Conference on Information and Knowledge Management.2016:679-688.

相关文章 15

[1]	郝敬宇, 文静轩, 刘华锋, 景丽萍, 于剑. 结合全局信息的深度图解耦协同过滤 Deep Disentangled Collaborative Filtering with Graph Global Information 计算机科学, 2023, 50(1): 41-51. https://doi.org/10.11896/jsjkx.220900255
[2]	程章桃, 钟婷, 张晟铭, 周帆. 基于图学习的推荐系统研究综述 Survey of Recommender Systems Based on Graph Learning 计算机科学, 2022, 49(9): 1-13. https://doi.org/10.11896/jsjkx.210900072
[3]	王冠宇, 钟婷, 冯宇, 周帆. 基于矢量量化编码的协同过滤推荐方法 Collaborative Filtering Recommendation Method Based on Vector Quantization Coding 计算机科学, 2022, 49(9): 48-54. https://doi.org/10.11896/jsjkx.210700109
[4]	秦琪琦, 张月琴, 王润泽, 张泽华. 基于知识图谱的层次粒化推荐方法 Hierarchical Granulation Recommendation Method Based on Knowledge Graph 计算机科学, 2022, 49(8): 64-69. https://doi.org/10.11896/jsjkx.210600111
[5]	方义秋, 张震坤, 葛君伟. 基于自注意力机制和迁移学习的跨领域推荐算法 Cross-domain Recommendation Algorithm Based on Self-attention Mechanism and Transfer Learning 计算机科学, 2022, 49(8): 70-77. https://doi.org/10.11896/jsjkx.210600011
[6]	帅剑波, 王金策, 黄飞虎, 彭舰. 基于神经架构搜索的点击率预测模型 Click-Through Rate Prediction Model Based on Neural Architecture Search 计算机科学, 2022, 49(7): 10-17. https://doi.org/10.11896/jsjkx.210600009
[7]	齐秀秀, 王佳昊, 李文雄, 周帆. 基于概率元学习的矩阵补全预测融合算法 Fusion Algorithm for Matrix Completion Prediction Based on Probabilistic Meta-learning 计算机科学, 2022, 49(7): 18-24. https://doi.org/10.11896/jsjkx.210600126
[8]	蔡晓娟, 谭文安. 一种改进的融合相似度和信任度的协同过滤算法 Improved Collaborative Filtering Algorithm Combining Similarity and Trust 计算机科学, 2022, 49(6A): 238-241. https://doi.org/10.11896/jsjkx.210400088
[9]	何亦琛, 毛宜军, 谢贤芬, 古万荣. 基于点割集图分割的矩阵变换与分解的推荐算法 Matrix Transformation and Factorization Based on Graph Partitioning by Vertex Separator for Recommendation 计算机科学, 2022, 49(6A): 272-279. https://doi.org/10.11896/jsjkx.210600159
[10]	洪志理, 赖俊, 曹雷, 陈希亮, 徐志雄. 基于遗憾探索的竞争网络强化学习智能推荐方法研究 Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration 计算机科学, 2022, 49(6): 149-157. https://doi.org/10.11896/jsjkx.210600226
[11]	郭亮, 杨兴耀, 于炯, 韩晨, 黄仲浩. 基于注意力机制和门控网络相结合的混合推荐系统 Hybrid Recommender System Based on Attention Mechanisms and Gating Network 计算机科学, 2022, 49(6): 158-164. https://doi.org/10.11896/jsjkx.210500013
[12]	熊中敏, 舒贵文, 郭怀宇. 融合用户偏好的图神经网络推荐模型 Graph Neural Network Recommendation Model Integrating User Preferences 计算机科学, 2022, 49(6): 165-171. https://doi.org/10.11896/jsjkx.210400276
[13]	余皑欣, 冯秀芳, 孙静宇. 结合物品相似性的社交信任推荐算法 Social Trust Recommendation Algorithm Combining Item Similarity 计算机科学, 2022, 49(5): 144-151. https://doi.org/10.11896/jsjkx.210300217
[14]	陈壮, 邹海涛, 郑尚, 于化龙, 高尚. 基于用户覆盖及评分差异的多样性推荐算法 Diversity Recommendation Algorithm Based on User Coverage and Rating Differences 计算机科学, 2022, 49(5): 159-164. https://doi.org/10.11896/jsjkx.210300263
[15]	任胜兰, 郭慧娟, 黄文豪, 汤志宏, 亓慧. 基于注意力机制交互卷积神经网络的推荐方法 Recommendation Method Based on Attention Mechanism Interactive Convolutional Neural Network 计算机科学, 2022, 49(10): 126-131. https://doi.org/10.11896/jsjkx.220700064

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed