计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (12): 242-249.doi: 10.11896/jsjkx.231000057

• 人工智能 • 上一篇    下一篇

基于多模态双协同Gather Transformer网络的虚假信息检测方法

向旺1, 王金光2, 王一飞1, 钱胜胜3   

  1. 1 郑州大学河南先进技术研究院 郑州 450000
    2 合肥工业大学计算机与信息学院 合肥 230601
    3 中国科学院自动化研究所多模态人工智能系统全国重点实验室 北京 100190
  • 收稿日期:2023-10-10 修回日期:2024-03-09 出版日期:2024-12-15 发布日期:2024-12-10
  • 通讯作者: 钱胜胜(shengsheng.qian@nlpr.ia.ac.cn)
  • 作者简介:(1584462772@qq.com)
  • 基金资助:
    国家自然科学基金(62276257)

Multi-modal Dual Collaborative Gather Transformer Network for Fake News Detection

XIANG Wang1, WANG Jinguang2, WANG Yifei1, QIAN Shengsheng3   

  1. 1 Henan Institute of Advanced Technology, Zhengzhou University, Zhengzhou 450000, China
    2 School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China
    3 State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
  • Received:2023-10-10 Revised:2024-03-09 Online:2024-12-15 Published:2024-12-10
  • About author:XIANG Wang,born in 1996,postgra-duate.His main research interests include natural language processing and multimedia computing analysis.
    QIAN Shengsheng,born in 1991,Ph.D,associate professor.His main research interests include data mining and multimedia content analysis.
  • Supported by:
    National Natural Science Foundation of China(62276257).

PDF (PC)

151

摘要： 社交媒体网站是人们在日常生活中分享信息、表达和交换意见的便捷平台。随着用户数量的不断增加,社交媒体网站上出现了大量的信息数据。然而,由于用户没有检查共享信息的可靠性,这些信息的真实性难以保证,从而导致大量虚假信息在社交媒体上广泛传播。然而,现有方法大多存在以下局限性:1)大多数方法通过简单提取文本与视觉特征,将其拼接后得到多模态特征来进行虚假信息判断,忽略了模态间和模态内细粒度内在联系,缺乏对关键信息的检索和筛选;2)多模态信息间缺乏指导性的特征提取,文本和视觉等特征之间缺乏交互增强,对多模态信息的理解不足。为了应对这些挑战,提出了一种新颖的基于多模态双协同Gather Transformer网络(Multimodal Dual-Collaborative Gather Transformer Network,MDCGTN)的虚假信息检测方法。在MDCGTN模型中,通过文本-视觉编码网络对文本和视觉信息的特征表示进行提取,将获得的视觉和文本特征表示输入多模态Gather Transformer网络进行多模态信息融合,使用Gather机制提取关键信息,充分捕捉和融合模态内和模态间细粒度关系。此外,设计了一个双协同机制对社交媒体帖子的多模态信息进行整合,以实现模态之间信息的交互和增强。在两个公开可用的基准数据集上进行了大量实验,结果表明,与现有的先进基准方法相比,所提方法准确率明显提升,证明了其对于虚假信息检测的优越性能。

关键词: 多模态, 虚假信息检测, 社交媒体, Gather Transformer网络

Abstract: Social media platforms are convenient platforms for people to share information,express opinions,and exchange ideas in their daily lives.With the increasing number of users,a large amount of data has emerged on social media websites.However,the authenticity of the shared information is difficult to be guaranteed due to users’ lack of verification.This situation has led to the widespread dissemination of a large amount of fake news on social media.However,existing methods suffer from the follo-wing limitations:1)Most existing methods rely on simple text and visual feature extraction,concatenating them to obtain multimodal features for detecting fake news,while ignoring the fine-grained intrinsic connections within and between modalities,and lacking retrieval and filtering of key information.2)There is a lack of guided feature extraction among multimodal information,with insufficient interaction and understanding between textual and visual features.To address these challenges,a novel multimodal dual-collaborative gather transformer network(MDCGTN) is proposed to overcome the limitations of existing methods.In the MDCGTN model,textual and visual features are extracted using a text-visual encoding network,and the obtained features are input into a multimodal gather transformer network for multimodal information fusion.The gathering mechanism is used to extract key information,fully capturing and fusing fine-grained relationships within and between modalities.In addition,a dual-collaborative mechanism is designed to integrate multimodal information in social media posts,enhancing interaction and understanding between modalities.Extensive experiments are conducted on two publicly available benchmark datasets.Compared to existing state-of-the-art benchmark methods,the proposed MDCGTN method achieves significant improvement in accuracy,demonstrating its superior performance in detecting fake news.

Key words: Multi-modal, Fake news detection, Social media, Gather transformer network

中图分类号: 

  • TP389
[1]CASTILLO C,MENDOZA M,POBLETE B.Information credibility on twitter[C]//Proceedings of the 20th International Conference on World Wide Web.2011:675-684.
[2]KWON S,CHA M,JUNG K,et al.Prominent features of rumor propagation in online social media[C]//2013 IEEE 13th International Conference on Data Mining.IEEE,2013:1103-1108.
[3]LIU X,NOURBAKHSH A,LI Q,et al.Real-time rumor debunking on twitter[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management.2015:1867-1870.
[4]MA J,GAO W,WEI Z,et al.Detect rumors using time series of social context information on microblogging websites[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management.2015:1751-1754.
[5]MA J,GAO W,MITRA P,et al.Detecting rumors from micro-blogs with recurrent neural networks[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.2016:3818-3824.
[6]YU F,LIU Q,WU S,et al.A Convolutional Approach for Misinformation Identification[C]//IJCAI.2017:3901-3907.
[7]WANG J,WANG Y C,HUANG M J.False information in social networks:Definition,detection and control[J].Computer Science,2021,48:263-277.
[8]HAO X,MING L.Deepfake Video Detection Based on 3D Con-volutional Neural Networks[J].Computer Science,2021,48(7):86-92.
[9]PROCTER R,CRUMP J,KARSTEDT S,et al.Reading theriots:What were the police doing on Twitter?[J].Policing and Society,2013,23(4):413-436.
[10]WANG J,MAO H,LI H.FMFN:Fine-grained multimodal fusion networks for fake news detection[J].Applied Sciences,2022,12(3):1093.
[11]QIAN S S,ZHANG T Z,XU C S.Survey of Multimedia Social Events Analysis[J].Computer Science,2021,48(3):97-112.
[12]WU X K,ZHAO T F.Application of natural language proces-sing in social communication:A review and future perspectives[J].Computer Science,2020,47(6):184-193.
[13]HAN Z M,ZHENG C Y,DUAN D G,et al.Associated Users Mining Algorithm Based on Multi-information Fusion Representation Learning[J].Computer Science,2019,46(4):77-82.
[14]KHATTAR D,GOUD J S,GUPTA M,et al.Mvae:Multimodal variational autoencoder for fake news detection[C]//The World Wide Web Conference.2019:2915-2921.
[15]SHANG L,KOU Z,ZHANG Y,et al.A duo-generative ap-proach to explainable multimodal covid-19 misinformation detection[C]//Proceedings of the ACM Web Conference.2022:3623-3631.
[16]LIN H,MA J,CHENG M,et al.Rumor detection on twitter with claim-guided hierarchical graph attention networks[J].arXiv:2110.04522,2021.
[17]CHEN M,CHU X,SUBBALAKSHMI K P.MMCoVaR:multimodal COVID-19 vaccine focused data repository for fake news detection and a baseline architecture for classification[C]//Proceedings of the 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining.2021:31-38.
[18]ZHOU X,MULAY A,FERRARA E,et al.Recovery:A multimodal repository for covid-19 news credibility research[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management.2020:3205-3212.
[19]SINGHAL S,KABRA A,SHARMA M,et al.Spotfake+:Amultimodal framework for fake news detection via transfer learning(student abstract)[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:13915-13916.
[20]LIU J,FENG K,PAN J Z,et al.MSRD:Multimodal Web Rumor Detection Method[J/OL].https://www.researchgate.net/publication/347001640.
[21]WANG Y,MA F,JIN Z,et al.Eann:Event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM Sigkdd International Conference on Knowledge Discovery & Data Mining.2018:849-857.
[22]YAO L,MAO C,LUO Y.Graph convolutional networks for text classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019:7370-7377.
[23]ZHOU X,WU J,ZAFARANI R.Safe:similarity-aware multi-modal fake news detection[C]//Advances in Knowledge Disco-very and Data Mining:24th Pacific-Asia Conference.2020:354-367.
[24]WEI Z X,LIANG J M.Design of Image Retrieval System Based on Speech Recognition[J].Applied Mechanics and Materials,2012,220:2371-2374.
[25]XUE J,WANG Y,TIAN Y,et al.Detecting fake news by exploring the consistency of multimodal data[J].Information Processing & Management,2021,58(5):102610.
[26]XU K,BA J,KIROS R,et al.Show,attend and tell:Neuralimage caption generation with visual attention[C]//InternationalConference on Machine Learning.PMLR,2015:2048-2057.
[27]BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv:1409.0473,2014.
[28]CHEN J,ZHANG H,HE X,et al.Attentive collaborative filtering:Multimedia recommendation with item-and component-le-vel attention[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval.2017:335-344.
[29]WANG S,HU L,CAO L,et al.Attention-based transactionalcontext embedding for next-item recommendation[C]//Procee-dings of the AAAI Conference on Artificial Intelligence.2018.
[30]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Confe-rence on Neural Information Processing Systems.2017:6000-6010.
[31]BAEVSKI A,AULI M.Adaptive input representations for neural language modeling[J].arXiv:1809.10853,2018.
[32]DAI Z,YANG Z,YANG Y,et al.Transformer-xl:Attentivelanguage models beyond a fixed-length context[J].arXiv:1901.02860,2019.
[33]RAE J W,POTAPENKO A,JAYAKUMAR S M,et al.Compressive transformers for long-range sequence modelling[J].arXiv:1911.05507,2019.
[34]RADFORD A,NARASIMHAN K,SALIMANS T,et al.Improving language understanding by generative pre-training[J/OL].https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
[35]DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training of deep bidirectional transformers for language understanding[J].arXiv:1810.04805,2018.
[36]CHEN T,LI X,YIN H,et al.Call attention to rumors:Deep attention based recurrent neural networks for early rumor detection[C]//Trends and Applications in Knowledge Discovery and Data Mining.Springer International Publishing,2018:40-52.
[37]GUO H,CAO J,ZHANG Y,et al.Rumor detection with hierarchical social attention network[C]//Proceedings of the 27th ACM International Conference on Information and Knowledge Management.2018:943-951.
[38]KOU Z,ZHANG D Y,SHANG L,et al.Exfaux:A weakly supervised approach to explainable fauxtography detection[C]//2020 IEEE International Conference on Big Data(Big Data).IEEE,2020:631-636.
[39]SHU K,CUI L,WANG S,et al.defend:Explainable fake news detection[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.2019:395-405.
[40]ZHANG W,GUI L,HE Y.Supervised contrastive learning for multimodal unreliable news detection in covid-19 pandemic[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.2021:3637-3641.
[41]SINGHAL S,SHAH R R,CHAKRABORTY T,et al.Spot-fake:A multi-modal framework for fake news detection[C]//2019 IEEE fifth International Conference on Multimedia Big Data(BigMM).IEEE,2019:39-47.
[42]WANG J,QIAN S,HU J,et al.Positive Unlabeled Fake News Detection Via Multi-Modal Masked Transformer Network[J].IEEE Transactions on Multimedia,2023,26:234-244.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发