Computer Science ›› 2024, Vol. 51 ›› Issue (11): 39-46.doi: 10.11896/jsjkx.240700062

• Social Media Fake News Detection • Previous Articles     Next Articles

Multimodal Adaptive Fusion Based Detection of Fake News in Short Videos

ZHU Feng1, ZHANG Tinghui1, LI Peng1,2, XU He1,2   

  1. 1 College of Computer Science,Nanjing University of Posts and Telecommunications,Nanjing 210023,China
    2 Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks,Nanjing 210023,China
  • Received:2024-07-10 Revised:2024-08-29 Online:2024-11-15 Published:2024-11-06
  • About author:ZHU Feng,born in 1987,Ph.D,assistant professor,master supervisor.His main research interests include cyberspace security,Internet of Things security,and operating system security.
    LI Peng,born in 1979,Ph.D,professor,Ph.D supervisor,is a member of CCF(No.48573M).His main research interests include computer communication networks,clouding computing,and information security.
  • Supported by:
    National Natural Science Foundation of China(61902196,62102196),Scientific and Technological Support Project of Jiangsu Province(BE2019740) and Six Talent Peaks Project of Jiangsu Province(RJFW-111).

Abstract: With the rapid development of Internet and social media,the dissemination route of news is no longer limited to traditional media channels.Semantically rich multimodal data becomes the carrier of news while fake news has been widely spread.As the proliferation of false news will have an unpredictable impact on individuals and society,the detection of false news has become a current research hotspot.Existing multimodal false news detection methods only focus on text and image data,which not only fail to fully utilize the multimodal information in short videos but also ignore the consistency and difference features between different modalities.As a result,it is difficult for them to give full play to the advantages of multimodal fusion.To solve this pro-blem,a fake news detection model for short videos based on multimodal adaptive fusion is proposed.This model extracts features from multimodal data in short videos,uses cross-modal alignment fusion to obtain the consistency and complementarity features among different modalities,and then achieves adaptive fusion based on the contribution of different modal features to the final fusion result.Finally,a classifier is used to achieve fake news detection.The results of the experiment conducted on a publicly avai-lable short video dataset demonstrate that the accuracy,precision,recall,and F1-score of the proposed model are higher than those of the state-of-the-art models.

Key words: Fake news detection, Multimodal, Short video, Cross-modal fusion, Adaptive fusion

CLC Number: 

  • TP391
[1] OLAN F,JAYAWICKRAMA U,ARAKPOGUN E O,et al.Fake News on Social Media:The Impact on Society[J].Information Systems Frontiers,2024,26(2):443-458.
[2] TUFCHI S,YADAV A,AHMED T.A Comprehensive Survey of Multimodal Fake News Detection Techniques:Advances,Challenges,and Opportunities[J].International Journal of Multi-media Information Retrieval,2023,12(2):28.
[3] ZHANG X,GHORBANI A A.An Overview of Online FakeNews:Characterization,Detection,and Discussion[J].Information Processing & Management,2020,57(2):102025.
[4] VOSOUGHI S,ROY D,ARAL S.The Spread of True and False News Online[J].Science,2018,359(6380):1146-1151.
[5] XU K,WANG F,WANG H,et al.Detecting Fake News over Online Social Media via Domain Reputations and Content Understanding[J].Tsinghua Science and Technology,2020,25(1):20-27.
[6] LI X,LI S,LI J,et al.Detection of Fake-video Uploaders on So-cial Media Using Naive Bayesian Model with Social Cues[J].Scientific Reports,2021,11(1):16068.
[7] LIAO Q,CHAI H,HAN H,et al.An Integrated Multi-taskModel for Fake News Detection[J].IEEE Transactions on Knowledge and Data Engineering,2022,34(11):5154-5165.
[8] CAO J,QI P,SHENG Q,et al.Exploring the Role of VisualContent in Fake News Detection[J].arXiv:2003.05096,2020.
[9] RAJ C,MEEL P.Convnet Frameworks for Multi-modal Fake News Detection[J].Applied Intelligence,2021,51(11):8132-8148.
[10] AMRI S,SALLAMI D,AÏMEUR E.Exmulf:An ExplainableMultimodal Content-based Fake News Detection System[C]//14th International Symposium on Foundations and Practice of Security.2021:177-187.
[11] WANG H,GONG L,ZHOU Z,et al.Detecting Mis/Dis-information from Social Media with Semantic Enhancement[J].Data Analysis and Knowledge Discovery,2023,7(2):48-60.
[12] YU F,LIU Q,WU S,et al.A Convolutional Approach for Misinformation Identification[C]//26th International Joint Confe-rence on Artificial Intelligence.2017:3901-3907.
[13] MOHTARAMI M,BALY R,GLASS J,et al.Automatic Stance Detection Using End-to-End Memory Networks[C]//2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.2018:767-776.
[14] MA J,GAO W,WONG K.Detect Rumors on Twitter by Promoting Information Campaigns with Generative Adversarial Learning[C]//2019 The World Wide Web Conference.2019:3049-3055.
[15] DOU Y,SHU K,XIA C,et al.User Preference-aware FakeNews Detection[C]//44th International ACM SIGIRConfe-rence on Research and Development in Information Retrieval.2021:2051-2055.
[16] KAUSAR N,ALIKHAN A,SATTAR M.Towards Better Representation Learning Using Hybrid Deep Learning Model for Fake News Detection[J].Social Network Analysis and Mining,2022,12(1):165.
[17] YU W,GE J,YANG Z,et al.Multi-domain Fake News Detec-tion for History News Environment Perception[C]//17th IEEE Conference on Industrial Electronics and Applications.2022:428-433.
[18] SONG C,TENG Y,ZHU Y,et al.Dynamic Graph Neural Network for Fake News detection[J].Neurocomputing,2022,505:362-374.
[19] QI P,CAO J,YANG T,et al.Exploiting Multi-domain Visual Information for Fake News Detection[C]//2019 IEEE International Conference on Data Mining.2019:518-527.
[20] LIANG Y,TOHTI T,HAMDULLA A.Multi-modal False Information Detection via Multi-layer CNN-based Feature Fusion and Multi-classifier Hybrid Prediction[J].Computer Enginee-ring and Science,2023,45(6):1087-1096.
[21] YE Z,LUO S,YU J.Multimodal Misinformation DetectionModel with Social Network Graph[J].Application Research of Computers,2024,41(7):1-8.
[22] QU Z,MENG Y,MUHAMMAD G,et al.QMFND:A Quantum Multimodal Fusion-based Fake News Detection Model for Social Media[J].Information Fusion,2024,104:102172.
[23] REZAYI S,SOLEYMANI S,ARABNIA H R,et al.SociallyAware Multimodal Deep Neural Networks for Fake News Classification[C]//4th International Conference on Multimedia Information Processing and Retrieval.2021:253-259.
[24] GÔLO M P S,DE SOUZA M C,ROSSI R G,et al.One-class Learning for Fake News Detection through Multimodal Variational Autoencoders[J].Engineering Applications of Artificial Intelligence,2023,122:106088.
[25] SILVA A,LUO L,KARUNASEKERA S,et al.Embracing Domain Differences in Fake News:Cross-domain Fake News Detection Using Multi-modal Data[C]//35th AAAI Conference on Artificial Intelligence.2021:557-565.
[26] MOSALLANEZHAD A,KARAMI M,SHU K,et al.DomainAdaptive Fake News Detection via Reinforcement Learning[C]//31st ACM World Wide Web Conference.2022:3632-3640.
[27] HOU R,PÉREZ-ROSAS V,LOEB S,et al.Towards Automatic Detection of Misinformation in Online Medical Videos[C]//21st International Conference on Multimodal Interaction.2019:235-243.
[28] SERRANO J C M,PAPAKYRIAKOPOULOS O,HEGELICHS.NLP-based Feature Extraction for The Detection of Covid-19 Misinformation Videos on YouTube[C]//1st Workshop on NLP for COVID-19 at the 58th Annual Meeting of the Association for Computational Linguistics.2020:1-7.
[29] CHOI H,KO Y.Using Topic Modeling and Adversarial Neural Networks for Fake News Video Detection[C]//30th ACM International Conference on Information & Knowledge Management.2021:2950-2954.
[30] SHANG L,KOU Z,ZHANG Y,et al.A Multimodal Misinformation Detector for Covid-19 Short Videos on TikTok[C]//2021 IEEE International Conference on Big Data.2021:899-908.
[31] WANG W Y.“Liar,Liar Pants on Fire”:A New Benchmark Dataset for Fake News Detection[C]//55th Annual Meeting of the Association for Computational Linguistics.2017:422-426.
[32] SHAHI G K,DIRKSON A,MAJCHRZAK T A.An Exploratory Study of Covid-19 Misinformation on Twitter[J].Online Social Networks and Media,2021,22:100104.
[33] KOCHKINA E,HOSSAIN T,LOGAN R L,et al.Evaluatingthe Generalisability of Neural Rumour Verification Models[J].Information Processing & Management,2023,60(1):103116.
[34] BOIDIDOU C,PAPADOPOULOS S,ZAMPOGLOU M,et al.Detection and Visualization of Misleading Content on Twitter[J].International Journal of Multimedia Information Retrieval,2018,7(1):71-86.
[35] JIN Z,CAO J,GUO H,et al.Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]//25th ACM International Conference on Multimedia.2017:795-816.
[36] NAKAMURA K,LEVY S,WANG W Y.Fakeddit:A NewMultimodal Benchmark Dataset for Fine-Grained Fake News Detection[C]//12th Language Resources and Evaluation Conference.2020:6149-6157.
[37] JINDAL S,SOOD R,SINGH R,et al.Newsbag:A Multimodal Benchmark Dataset for Fake News Detection[C]//The AAAI-20 Workshop on Artificial Intelligence Safety.2020:138-145.
[38] GAO G,FANG Y,HAN Y,et al.Construction of Multi-modal Social Media Dataset for Fake News Detection[J].Chinese Journal of Network and Information Security,2023,9(4):144-154.
[39] ZHOU X,MULAY A,FERRARA E,et al.Recovery:A Multimodal Repository for Covid-19 News Credibility Research[C]//29th ACM International Conference on Information & Know-ledge Management.2020:3205-3212.
[40] QI P,BU Y,CAO J,et al.FakeSV:A Multimodal Benchmarkwith Rich Social Context for Fake News Detection on Short Vi-deo Platforms[C]//37th AAAI Conference on Artificial Intelligence.2023:14444-14452.
[41] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is All You Need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.2017:6000-6010.
[42] BAI C,CHEN H,KUMAR S,et al.M2p2:Multimodal Persuasion Prediction Using Adaptive Fusion[J].IEEE Transactions on Multimedia,2021,25:942-952.
[43] HERSHEY S,CHAUDHURI S,ELLIS D P W,et al.CNN Architectures for Large-Scale Audio Classification[C]//2017 IEEE International Conference on Acoustics,Speech and Signal Processing.2017:131-135.
[44] LUO W.Research and Implementation of Text Topic Classification Based on Text CNN[C]//3rd International Conference on Computer Vision,Image and Deep Learning & International Conference on Computer Engineering and Applications.2022:1152-1155.
[45] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-scale Image Recognition[C]//3rd International Conference on Learning Representations.2015:1-14.
[1] ZHANG Tianzhi, ZHOU Gang, LIU Hongbo, LIU Shuo, CHEN Jing. Text-Image Gated Fusion Mechanism for Multimodal Aspect-based Sentiment Analysis [J]. Computer Science, 2024, 51(9): 242-249.
[2] MO Shuyuan, MENG Zuqiang. Multimodal Sentiment Analysis Model Based on Visual Semantics and Prompt Learning [J]. Computer Science, 2024, 51(9): 250-257.
[3] LU Xulin, LI Zhihua. IoT Device Recognition Method Combining Multimodal IoT Device Fingerprint and Ensemble Learning [J]. Computer Science, 2024, 51(9): 371-382.
[4] WANG Chao, TANG Chao, WANG Wenjian, ZHANG Jing. Infrared Human Action Recognition Method Based on Multimodal Attention Network [J]. Computer Science, 2024, 51(8): 232-241.
[5] YAN Qiuyan, SUN Hao, SI Yuqing, YUAN Guan. Multimodality and Forgetting Mechanisms Model for Knowledge Tracing [J]. Computer Science, 2024, 51(7): 133-139.
[6] LI Jiaying, LIANG Yudong, LI Shaoji, ZHANG Kunpeng, ZHANG Chao. Study on Algorithm of Depth Image Super-resolution Guided by High-frequency Information ofColor Images [J]. Computer Science, 2024, 51(7): 197-205.
[7] YIN Baosheng, ZHOU Peng. Chinese Medical Named Entity Recognition with Label Knowledge [J]. Computer Science, 2024, 51(6A): 230500203-7.
[8] WANG Yifan, ZHANG Xuefang. Modality Fusion Strategy Research Based on Multimodal Video Classification Task [J]. Computer Science, 2024, 51(6A): 230300212-5.
[9] HUANG Yumin, ZHAO Chanchan. Study on Three-level Short Video User Portrait Based on Improved Topic Model Method [J]. Computer Science, 2024, 51(6A): 230800093-7.
[10] YU Bihui, TAN Shuyue, WEI Jingxuan, SUN Linzhuang, BU Liping, ZHAO Yiman. Vision-enhanced Multimodal Named Entity Recognition Based on Contrastive Learning [J]. Computer Science, 2024, 51(6): 198-205.
[11] DUAN Yuxiao, HU Yanli, GUO Hao, TAN Zhen, XIAO Weidong. Study on Improved Fake Information Detection Method Based on Cross-modal CorrelationAmbiguity Learning [J]. Computer Science, 2024, 51(4): 307-313.
[12] ZHENG Cheng, SHI Jingwei, WEI Suhua, CHENG Jiaming. Dual Feature Adaptive Fusion Network Based on Dependency Type Pruning for Aspect-basedSentiment Analysis [J]. Computer Science, 2024, 51(3): 205-213.
[13] LI Jianqiu, LIU Wanping, HUANG Dong, ZHANG Qiong. Multimodal Fusion Based Dynamic Malware Detection [J]. Computer Science, 2024, 51(11A): 240200098-7.
[14] ZHU Xudong, LAI Teng. Multimodal Contrastive Learning Based Scene Graph Generation [J]. Computer Science, 2024, 51(11A): 231200185-5.
[15] ZHOU Yan, XU Yewen, PU Lei, XU Xuemiao, LIU Xiangyu, ZHOU Yuexia. Research Progress of Image 3D Object Detection in Autonomous Driving Scenario [J]. Computer Science, 2024, 51(11): 133-147.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!