跨模态特征融合与对齐的虚假信息检测模型

doi:10.11896/jsjkx.250700054

摘要/Abstract

摘要： 虚假信息通常以夸大、歪曲或误导性的陈述进行传播,进而塑造负面社会舆论,严重危害公共安全。当前虚假信息通常是多模态的,现有检测方法往往分别提取各模态特征后进行融合,忽略了模态间的相关性,难以全面捕获细节及相关性信息,导致检测性能不够理想。针对这些问题,提出了基于跨模态特征融合与对齐的检测模型(CMFFA)。CMFFA优化了特征提取、融合和分类的模式,从宏观和微观两个角度提取模态特征,通过注意力机制进行特征增强,并通过计算模态间相似度评估模态间歧义性,以自适应地进行跨模态特征融合。首先,采用预训练模型编码文本和图像的单模态特征和跨模态特征;然后,在进行跨模态特征融合时,通过模态间歧义性分析,自适应地调整跨模态特征的使用比例,以更好地实现跨模态特征的融合,进而提升虚假信息检测性能。在公开的中英文虚假信息数据集上与已有虚假信息检测方法进行对比,实验结果表明,所提模型在F1值、精确率、召回率上均有明显提升,验证了其有效性和优越性。

关键词: 虚假信息检测, 跨模态特征融合, 模态间歧义性分析, 注意力机制

Abstract: Fake news is usually spread by exaggerated,distorted or misleading statements,which can shape negative social opinion and seriously endanger public safety.The current fake news is usually multi-modal,and the existing detection methods often fuse the features of each modality after extracting them separately,ignoring the correlation between modalities,which makes it difficult to fully capture the details and correlation information,resulting in unsatisfactory detection performance.To solve these pro-blems,this paper proposes a detection model based on cross-modal feature fusion and alignment(CMFFA).CMFFA optimizes the modes of feature extraction,fusion and classification,extracts modal features from macro and micro perspectives,enhances features through the attention mechanism,and evaluates the ambiguity between modalities by calculating the similarity between modalities,so as to adaptively perform cross-modal feature fusion.Firstly,the pre-trained model is used to encode the single-modal features and cross-modal features of text and image.Then,in the cross-modal feature fusion,the proportion of cross-modal features is adaptively adjusted through the ambiguity analysis between modalities,so as to better realize the fusion of cross-modal features and improve the performance of fake news detection.Experimental results show that,compared with the existing fake news detection methods on the public Chinese and English fake news datasets,the proposed model significantly improves the F1 value,precision and recall rate,verifying its effectiveness and superiority.

Key words: Fake news detection, Cross-modal feature fusion, Inter-modal ambiguity analysis, Attention mechanism

中图分类号:

TP391

杨晨光, 卢记仓, 郭嘉兴. 跨模态特征融合与对齐的虚假信息检测模型[J]. 计算机科学, 2026, 53(8): 257-265. https://doi.org/10.11896/jsjkx.250700054

YANG Chenguang, LU Jicang, GUO Jiaxing. Fake News Detection Model Based on Cross-modal Feature Fusion and Alignment[J]. Computer Science, 2026, 53(8): 257-265. https://doi.org/10.11896/jsjkx.250700054

参考文献

[1] HU Z R,HUANG C X.China New Media Development Report No.15[M]//Beijing:Social Sciences Academic Press,2024.
[2] BONDIELLI A,MARCELLONI F.A Survey on Fake News and Rumor Detection Techniques[J].Information Sciences,2019,497:38-55.
[3] KUMAR K,GEETHAKUMARI G.Detecting Misinformationin Online Social Networks Using Cognitive Psychology[J].Human-centric Computing and Information Sciences,2014,4(1):14-26.
[4] TRUEMAN T E,KUMAR A,NARAYANASAMY P,et al.Attention-based C-BiLSTM for Fake News Detection[J].Applied Soft Computing,2021,110:107600.
[5] WANG W Y.“Liar,Liar Pants on Fire”:a New Benchmark Dataset for Fake News Detection[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Stroudsburg:ACL,2017:422-426.
[6] CHAWDA S,PATIL A,SINGH A,et al.A Novel Approach for Clickbait Detection[C]//Proceedings of the 3rd International Conference on Trends in Electronics and Informatics.2019:1318-1321.
[7] RAI N,KUMAR D,KAUSHIK N,et al.Fake news classification using transformer based enhanced LSTM and BERT[C]//International Journal of Cognitive Computing in Engineering.2022:98-105.
[8] AHN Y,JEONG C.Natural Language Contents Evaluation System for Detecting Fake News Using Deep Learning[C]//Proceedings of the 16th International Joint Conference on Computer Science and Software Engineering.IEEE,2019:289-292.
[9] DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologie.2019:4171-4186.
[10] YU F,LIU Q,WU S,et al.A Convolutional Approach for Misinformation Identification[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence.2017:3901-3907.
[11] RAZA N,ABDULKADIR S J,ABID Y A,et al.Enhancing fake news detection with transformer-based deep learning:A multidisciplinary approach[J].PLoS One,2025,20(9):e0330954.
[12] XU F,LI M H,HUANG Q,et al.Rumor Detection Model based on Graph Convolutional Neural Network Driven by Knowledge Graph[J].Science China Information Science,2023,53(4):663-681.
[13] LIANG X,ZHANG Q,SHI C,et al.MSynFD:Multi-hop Syntax Aware Fake News Detection[C]//Proceedings of the ACM on Web Conference.2024:4128-4137.
[14] JIN Z W,CAO J,GUO H,et al.Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia.2017:795-816.
[15] WANG Y,MA F,JIN Z,et al.EANN:Event Adversarial Neural Networks for Multi-Modal Fake News Detection[C]//Procee-dings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.2018:849-857.
[16] KIM Y.Convolutional Neural Networks for Sentence Classification[J].arXiv:1408.5882,2014.
[17] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[J].arXiv:1409.1556,2014.
[18] ZHANG H,FANG Q,QIAN S,et al.Multi-modal Knowledge-Aware Event Memory Network for Social Media Rumor Detection[C]//Proceedings of the 27th ACM Conference on Multimedia.2019:1942-1951.
[19] WU Y,ZHAN P,ZHANG Y,et al.Multimodal Fusion with Co-Attention Networks for Fake News Detection[C]//Proceedings of the Findings Association for Computational Linguistics.2021:2560-2569.
[20] QIAN S,WANG J,HU J,et al.Hierarchical Multi-Modal Contextual Attention Network for Fake News Detection[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.2021:153-162.
[21] SONG C,NING N W,ZHANG Y L,et al.A Multimodal Fake News Detection Model Based on Crossmodal Attention Residual And Multichannel Convolutional Neural Networks[J].Information Processing & Management,2021,58(1):102437.
[22] HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[23] LIU Y,WU Y F.Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent And Convolutional Networks[C]//Proceedings of the AAAI Confe-rence on Artificial Intelligence.2018:354-361.
[24] JING J,WU H,SUN J,et al.Multimodal Fake News Detection via Progressive Fusion Networks[J].Information Processing & Management,2023,60(1):103120.
[25] LIU Z,LIN Y,CAO Y,et al.Swin Transformer:Hierarchical Vision Transformer using Shifted Windows[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.2021:9992-10002.
[26] ZHOU X Y,WU J D,ZAFARANI R.SAFE:Similarity-Aware Multi-modal Fake News Detection[C]//Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining.2020:354-367.
[27] CHEN Y X,LI D S,ZHANG P,et al.Cross-modal Ambiguity Learning for Multimodal Fake News Detection[C]//Procee-dings of the ACM Web Conference.2022:2897-2905.
[28] ZHOU Y,YANG Y,YING Q,et al.Multimodal Fake News Detection via CLIP-guided Learning[C]//Proceedings of the IEEE International Conference on Multimedia and Expo.2023:2825-2830.
[29] WANG J,ZHANG H,LIU C,et al.Fake News Detection via Multi-scale Semantic Alignment and Cross-modal Attention[C]//Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.2024:2406-2410.
[30] KUMAR V.Generative Adversarial Networks-aided IntrusionDetection System[C]//Generative Adversarial Networks and Deep Learning.2024:79-98.
[31] HE P,ZHANG H,CAO S,et al.Cross-Modal Fake News Detection Method Based on Multi-Level Fusion Without Evidence[J].Algorithms,2025,18(7):426.
[32] LIU Y F,LIU Y K,LI Z L,et al.Modality Interactive Mixture-of-Experts for Fake News Detection[C]//Proceedings of the ACM on Web Conference 2025(WWW’25).2025:5139-5150.
[33] HU J,YANG M,TANG B B,et al.Integrating Message Content and Propagation Path for Enhanced False Information Detection Using Bidirectional Graph Convolutional Neural Networks[J].Applied Science,2025,15:3457.
[34] CHEN X Q,HUANG X Y,GAO Q,et al.Enhancing text-centric fake news detection via external knowledge distillation from LLMs[J].Neural networks:the official journal of the International Neural Network Society,2025,187:107377.
[35] RADFORD A,KIM J,HALLACY C,et al.Learning Transferable Visual Models from Natural Language Supervision[C]//Proceedings of the International Conference on Machine Lear-ning.2021:8748-8763.
[36] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Confe-rence on Neural Information Processing Systems.2017:6000-6010.
[37] VINOD N,GEOFFREY E H.Rectified Linear Units ImproveRestricted Boltzmann Machines Vinod Nair[C]//Proceedings of the 27th International Conference on Machine Learning.2010:807-814.
[38] MCCLELLAND J,RUMELHART D.Parallel Distributed Processing:Explorations in the Microstructure of Cognition:Psychological and Biological Models[M].MIT Press,1987.
[39] KULLBACK S,LEIBLER R A.On Information and Sufficiency[J].The Annals of Mathematical Statistics,1951,22(1):79-86.
[40] KINGMA D P,WELLING M.Auto-Encoding Variational Bayes[C]//Proceedings of the International Conference on Learning Representations.2014.
[41] ZUBIAGA A,LIAKATA M,PROCTER R.Exploiting Context for Rumour Detection in Social Media[M]//Lecture Notes in Computer Science.2017:109-123.
[42] SHU K,MAHUDESWARAN D,WANG S H,et al.FakeNewsNet:A data repository with news content,social context,and spatiotemporal information for studying fake news on social media[J].Big Data,2020,8(3):171-188.
[43] MADRY A,MAKELOV A,SCHMIDT L,et al.Towards Deep Learning Models Resistant to Adversarial Attacks[C]//Proceedings of the International Conference on Learning Representations.2017.
[44] YUAN C,MA Q,ZHOU W,et al.Jointly Embedding the Local and Global Relations of Heterogeneous Graph for Rumor Detection[C]//Proceedings of IEEE International Conference on Data Mining.2019:796-805.
[45] DIDRIK P K,JIMMY B.Adam:A Method for Stochastic Optimization[C]//Proceedings of the International Conference on Learning Representations Computer Science.2014.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed