计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (11): 39-46.doi: 10.11896/jsjkx.240700062

• 社交媒体虚假信息检测 • 上一篇    下一篇

基于多模态自适应融合的短视频虚假新闻检测

朱枫1, 张廷辉1, 李鹏1,2, 徐鹤1,2   

  1. 1 南京邮电大学计算机学院 南京 210023
    2 江苏省无线传感网高技术研究重点实验室 南京 210023
  • 收稿日期:2024-07-10 修回日期:2024-08-29 出版日期:2024-11-15 发布日期:2024-11-06
  • 通讯作者: 李鹏(lipeng@njupt.edu.cn)
  • 作者简介:(zhufeng@njupt.edu)
  • 基金资助:
    国家自然科学基金(61902196,62102196);江苏省科技支撑计划项目(BE2019740);江苏省六大人才高峰高层次人才项目(RJFW-111)

Multimodal Adaptive Fusion Based Detection of Fake News in Short Videos

ZHU Feng1, ZHANG Tinghui1, LI Peng1,2, XU He1,2   

  1. 1 College of Computer Science,Nanjing University of Posts and Telecommunications,Nanjing 210023,China
    2 Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks,Nanjing 210023,China
  • Received:2024-07-10 Revised:2024-08-29 Online:2024-11-15 Published:2024-11-06
  • About author:ZHU Feng,born in 1987,Ph.D,assistant professor,master supervisor.His main research interests include cyberspace security,Internet of Things security,and operating system security.
    LI Peng,born in 1979,Ph.D,professor,Ph.D supervisor,is a member of CCF(No.48573M).His main research interests include computer communication networks,clouding computing,and information security.
  • Supported by:
    National Natural Science Foundation of China(61902196,62102196),Scientific and Technological Support Project of Jiangsu Province(BE2019740) and Six Talent Peaks Project of Jiangsu Province(RJFW-111).

PDF (PC)

431

摘要： 随着互联网和社交媒体的迅速发展,新闻的传播途径不再局限于传统的媒体渠道。语义丰富的多模态数据成为新闻的载体,虚假新闻也随之得到了广泛的传播。由于虚假新闻的泛滥会对个人以及社会产生难以预估的影响,针对虚假新闻的检测已经成为目前的研究热点。现有的多模态虚假新闻检测方法仅针对文本和图像数据,无法充分利用短视频中的多模态信息,且忽略了不同模态间的一致性和差异性特征,难以充分发挥多种模态融合的优势。为解决该问题,提出一种基于多模态自适应融合的短视频虚假新闻检测模型。首先对短视频中多模态数据进行特征提取,采用跨模态对齐融合获取不同模态间的一致性和互补性特征;然后根据不同模态特征对最终融合结果的贡献实现自适应融合;最后利用分类器实现虚假新闻检测。在公开的短视频数据集上的实验结果表明,该模型的准确率、精确率、召回率和F1分数都高于当前的先进基线模型。

关键词: 虚假新闻检测, 多模态, 短视频, 跨模态融合, 自适应融合

Abstract: With the rapid development of Internet and social media,the dissemination route of news is no longer limited to traditional media channels.Semantically rich multimodal data becomes the carrier of news while fake news has been widely spread.As the proliferation of false news will have an unpredictable impact on individuals and society,the detection of false news has become a current research hotspot.Existing multimodal false news detection methods only focus on text and image data,which not only fail to fully utilize the multimodal information in short videos but also ignore the consistency and difference features between different modalities.As a result,it is difficult for them to give full play to the advantages of multimodal fusion.To solve this pro-blem,a fake news detection model for short videos based on multimodal adaptive fusion is proposed.This model extracts features from multimodal data in short videos,uses cross-modal alignment fusion to obtain the consistency and complementarity features among different modalities,and then achieves adaptive fusion based on the contribution of different modal features to the final fusion result.Finally,a classifier is used to achieve fake news detection.The results of the experiment conducted on a publicly avai-lable short video dataset demonstrate that the accuracy,precision,recall,and F1-score of the proposed model are higher than those of the state-of-the-art models.

Key words: Fake news detection, Multimodal, Short video, Cross-modal fusion, Adaptive fusion

中图分类号: 

  • TP391
[1] OLAN F,JAYAWICKRAMA U,ARAKPOGUN E O,et al.Fake News on Social Media:The Impact on Society[J].Information Systems Frontiers,2024,26(2):443-458.
[2] TUFCHI S,YADAV A,AHMED T.A Comprehensive Survey of Multimodal Fake News Detection Techniques:Advances,Challenges,and Opportunities[J].International Journal of Multi-media Information Retrieval,2023,12(2):28.
[3] ZHANG X,GHORBANI A A.An Overview of Online FakeNews:Characterization,Detection,and Discussion[J].Information Processing & Management,2020,57(2):102025.
[4] VOSOUGHI S,ROY D,ARAL S.The Spread of True and False News Online[J].Science,2018,359(6380):1146-1151.
[5] XU K,WANG F,WANG H,et al.Detecting Fake News over Online Social Media via Domain Reputations and Content Understanding[J].Tsinghua Science and Technology,2020,25(1):20-27.
[6] LI X,LI S,LI J,et al.Detection of Fake-video Uploaders on So-cial Media Using Naive Bayesian Model with Social Cues[J].Scientific Reports,2021,11(1):16068.
[7] LIAO Q,CHAI H,HAN H,et al.An Integrated Multi-taskModel for Fake News Detection[J].IEEE Transactions on Knowledge and Data Engineering,2022,34(11):5154-5165.
[8] CAO J,QI P,SHENG Q,et al.Exploring the Role of VisualContent in Fake News Detection[J].arXiv:2003.05096,2020.
[9] RAJ C,MEEL P.Convnet Frameworks for Multi-modal Fake News Detection[J].Applied Intelligence,2021,51(11):8132-8148.
[10] AMRI S,SALLAMI D,AÏMEUR E.Exmulf:An ExplainableMultimodal Content-based Fake News Detection System[C]//14th International Symposium on Foundations and Practice of Security.2021:177-187.
[11] WANG H,GONG L,ZHOU Z,et al.Detecting Mis/Dis-information from Social Media with Semantic Enhancement[J].Data Analysis and Knowledge Discovery,2023,7(2):48-60.
[12] YU F,LIU Q,WU S,et al.A Convolutional Approach for Misinformation Identification[C]//26th International Joint Confe-rence on Artificial Intelligence.2017:3901-3907.
[13] MOHTARAMI M,BALY R,GLASS J,et al.Automatic Stance Detection Using End-to-End Memory Networks[C]//2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.2018:767-776.
[14] MA J,GAO W,WONG K.Detect Rumors on Twitter by Promoting Information Campaigns with Generative Adversarial Learning[C]//2019 The World Wide Web Conference.2019:3049-3055.
[15] DOU Y,SHU K,XIA C,et al.User Preference-aware FakeNews Detection[C]//44th International ACM SIGIRConfe-rence on Research and Development in Information Retrieval.2021:2051-2055.
[16] KAUSAR N,ALIKHAN A,SATTAR M.Towards Better Representation Learning Using Hybrid Deep Learning Model for Fake News Detection[J].Social Network Analysis and Mining,2022,12(1):165.
[17] YU W,GE J,YANG Z,et al.Multi-domain Fake News Detec-tion for History News Environment Perception[C]//17th IEEE Conference on Industrial Electronics and Applications.2022:428-433.
[18] SONG C,TENG Y,ZHU Y,et al.Dynamic Graph Neural Network for Fake News detection[J].Neurocomputing,2022,505:362-374.
[19] QI P,CAO J,YANG T,et al.Exploiting Multi-domain Visual Information for Fake News Detection[C]//2019 IEEE International Conference on Data Mining.2019:518-527.
[20] LIANG Y,TOHTI T,HAMDULLA A.Multi-modal False Information Detection via Multi-layer CNN-based Feature Fusion and Multi-classifier Hybrid Prediction[J].Computer Enginee-ring and Science,2023,45(6):1087-1096.
[21] YE Z,LUO S,YU J.Multimodal Misinformation DetectionModel with Social Network Graph[J].Application Research of Computers,2024,41(7):1-8.
[22] QU Z,MENG Y,MUHAMMAD G,et al.QMFND:A Quantum Multimodal Fusion-based Fake News Detection Model for Social Media[J].Information Fusion,2024,104:102172.
[23] REZAYI S,SOLEYMANI S,ARABNIA H R,et al.SociallyAware Multimodal Deep Neural Networks for Fake News Classification[C]//4th International Conference on Multimedia Information Processing and Retrieval.2021:253-259.
[24] GÔLO M P S,DE SOUZA M C,ROSSI R G,et al.One-class Learning for Fake News Detection through Multimodal Variational Autoencoders[J].Engineering Applications of Artificial Intelligence,2023,122:106088.
[25] SILVA A,LUO L,KARUNASEKERA S,et al.Embracing Domain Differences in Fake News:Cross-domain Fake News Detection Using Multi-modal Data[C]//35th AAAI Conference on Artificial Intelligence.2021:557-565.
[26] MOSALLANEZHAD A,KARAMI M,SHU K,et al.DomainAdaptive Fake News Detection via Reinforcement Learning[C]//31st ACM World Wide Web Conference.2022:3632-3640.
[27] HOU R,PÉREZ-ROSAS V,LOEB S,et al.Towards Automatic Detection of Misinformation in Online Medical Videos[C]//21st International Conference on Multimodal Interaction.2019:235-243.
[28] SERRANO J C M,PAPAKYRIAKOPOULOS O,HEGELICHS.NLP-based Feature Extraction for The Detection of Covid-19 Misinformation Videos on YouTube[C]//1st Workshop on NLP for COVID-19 at the 58th Annual Meeting of the Association for Computational Linguistics.2020:1-7.
[29] CHOI H,KO Y.Using Topic Modeling and Adversarial Neural Networks for Fake News Video Detection[C]//30th ACM International Conference on Information & Knowledge Management.2021:2950-2954.
[30] SHANG L,KOU Z,ZHANG Y,et al.A Multimodal Misinformation Detector for Covid-19 Short Videos on TikTok[C]//2021 IEEE International Conference on Big Data.2021:899-908.
[31] WANG W Y.“Liar,Liar Pants on Fire”:A New Benchmark Dataset for Fake News Detection[C]//55th Annual Meeting of the Association for Computational Linguistics.2017:422-426.
[32] SHAHI G K,DIRKSON A,MAJCHRZAK T A.An Exploratory Study of Covid-19 Misinformation on Twitter[J].Online Social Networks and Media,2021,22:100104.
[33] KOCHKINA E,HOSSAIN T,LOGAN R L,et al.Evaluatingthe Generalisability of Neural Rumour Verification Models[J].Information Processing & Management,2023,60(1):103116.
[34] BOIDIDOU C,PAPADOPOULOS S,ZAMPOGLOU M,et al.Detection and Visualization of Misleading Content on Twitter[J].International Journal of Multimedia Information Retrieval,2018,7(1):71-86.
[35] JIN Z,CAO J,GUO H,et al.Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]//25th ACM International Conference on Multimedia.2017:795-816.
[36] NAKAMURA K,LEVY S,WANG W Y.Fakeddit:A NewMultimodal Benchmark Dataset for Fine-Grained Fake News Detection[C]//12th Language Resources and Evaluation Conference.2020:6149-6157.
[37] JINDAL S,SOOD R,SINGH R,et al.Newsbag:A Multimodal Benchmark Dataset for Fake News Detection[C]//The AAAI-20 Workshop on Artificial Intelligence Safety.2020:138-145.
[38] GAO G,FANG Y,HAN Y,et al.Construction of Multi-modal Social Media Dataset for Fake News Detection[J].Chinese Journal of Network and Information Security,2023,9(4):144-154.
[39] ZHOU X,MULAY A,FERRARA E,et al.Recovery:A Multimodal Repository for Covid-19 News Credibility Research[C]//29th ACM International Conference on Information & Know-ledge Management.2020:3205-3212.
[40] QI P,BU Y,CAO J,et al.FakeSV:A Multimodal Benchmarkwith Rich Social Context for Fake News Detection on Short Vi-deo Platforms[C]//37th AAAI Conference on Artificial Intelligence.2023:14444-14452.
[41] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is All You Need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.2017:6000-6010.
[42] BAI C,CHEN H,KUMAR S,et al.M2p2:Multimodal Persuasion Prediction Using Adaptive Fusion[J].IEEE Transactions on Multimedia,2021,25:942-952.
[43] HERSHEY S,CHAUDHURI S,ELLIS D P W,et al.CNN Architectures for Large-Scale Audio Classification[C]//2017 IEEE International Conference on Acoustics,Speech and Signal Processing.2017:131-135.
[44] LUO W.Research and Implementation of Text Topic Classification Based on Text CNN[C]//3rd International Conference on Computer Vision,Image and Deep Learning & International Conference on Computer Engineering and Applications.2022:1152-1155.
[45] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-scale Image Recognition[C]//3rd International Conference on Learning Representations.2015:1-14.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发