计算机科学 ›› 2021, Vol. 48 ›› Issue (5): 117-123.doi: 10.11896/jsjkx.200400057

所属专题: 大数据&数据科学 虚拟专题

• 数据库&大数据&数据科学 • 上一篇    下一篇

融合多模态信息的社交网络谣言检测方法

张少钦1, 杜圣东1,2,3, 张晓博1,2,3, 李天瑞1,2,3   

  1. 1 西南交通大学信息科学与技术学院 成都611756
    2 西南交通大学人工智能研究院 成都611756
    3 综合交通大数据应用技术国家工程实验室 成都611756
  • 收稿日期:2020-04-14 修回日期:2020-07-13 出版日期:2021-05-15 发布日期:2021-05-09
  • 通讯作者: 李天瑞(trli@swjtu.edu.cn)
  • 基金资助:
    国家重点研发计划 (2017YFB1401400)

Social Rumor Detection Method Based on Multimodal Fusion

ZHANG Shao-qin1, DU Sheng-dong1,2,3, ZHANG Xiao-bo1,2,3, LI Tian-rui1,2,3   

  1. 1 School of Information Science and Technology,Southwest Jiaotong University,Chengdu 611756,China
    2 Institute of Artificial Intelligence,Southwest Jiaotong University,Chengdu 611756,China
    3 National Engineering Laboratory of Integrated Transportation Big Data Application Technology,Chengdu 611756,China
  • Received:2020-04-14 Revised:2020-07-13 Online:2021-05-15 Published:2021-05-09
  • About author:ZHANG Shao-qin,born in 1994,postgraduate.Her main research interests include data mining and natural language processing.(zsq1024@163.com)
    LI Tian-rui,born in 1969,Ph.D,professor,Ph.D supervisor,is a distinguished member of China Computer Federation.His main research interests include big data intelligence,rough sets and granular computing.
  • Supported by:
    National Key R&D Program of China (2017YFB1401400).

摘要: 随着社交网络平台的发展,社交网络已经成为人们获取信息的重要来源。然而社交网络的便利性也导致了虚假谣言的快速传播。与纯文本的谣言相比,带有多媒体信息的网络谣言更容易误导用户以及被传播,因此对多模态的网络谣言检测在现实生活中有着重要意义。研究者们已提出若干多模态的网络谣言检测方法,但这些方法都没有充分挖掘出视觉特征和融合文本与视觉的联合表征特征。为弥补这些不足,提出了一个基于深度学习的端到端的多模态融合网络。该网络首先抽取出图片中各个兴趣区域的视觉特征,然后使用多头注意力机制将文本和视觉特征进行更新与融合,最后将这些特征进行基于注意力机制的拼接以用于社交网络多模态谣言检测。在推特和微博公开数据集上进行对比实验,结果表明,所提方法在推特数据集上F1值有13.4%的提升,在微博数据集上F1值有1.6%的提升。

关键词: 多模态, 目标检测, 深度学习, 谣言检测

Abstract: With the development of social networking platforms,social networks have become an important source of information for people.However,the convenience of social networks has also led to the rapid propagation of false rumors.Compared with textual rumors,social network rumors with multimedia content are more likely to mislead users and get dissemination,so the detection of multi-modal rumors is of great significance in real life.Several multi-modal rumor detection methods have been proposed,but the visual features and joint representation of text and visual features have not been fully explored in current approaches.To make up for these shortcomings,an end-to-end multi-modal fusion network based on deep learning is developed.Firstly,the visual features of each region of interest in the image are extracted.Then,the text and the visual features are updated and fused by using a multi-head attention mechanism.Finally,these features are concatenated based on the attention mechanism for the detection of multi-modal rumors in social networks.Comparative experiments on the public data sets of Twitter and Weibo are conducted and experimental results show that the proposed method has a 13.4% F1 value increase on Twitter data set and a 1.6% F1 value increase on Weibo data set.

Key words: Deep learning, Multi-moda, Object detection, Rumor detection

中图分类号: 

  • TP391
[1]CHEN Y F,LI Z Y,LIANG X,et al.Review on Rumor Detection of Online Social Networks[J].Chinese Journal of Compu-ters,2018,41(7):1648-1677.
[2]KARPATHY A,LI F F.Deep Visual-Semantic Alignments for Generating Image Descriptions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(4):664-676.
[3]ANDERSON P,HE X,BUEHLER C,et al.Bottom-Up andTop-Down Attention for Image Captioning and Visual Question Answering[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.ACM,2018:6077-6086.
[4]GAO P,JIANG Z,YOU H,et al.Dynamic fusion with intra-and inter-modality attention flow for visual question answering[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.ACM,2019:6632-6641.
[5]BALTRUSAITIS T,AHUJA C,MORENCY L P.Multimodal Machine Learning:A Survey and Taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(2):423-443.
[6]CASTILLO C,MENDOZA M,POBLETE B.Predicting infor-mation credibility in time-sensitive social media[J].Internet Research,2013,23(5):560-588.
[7]ZHAO Z,RESNICK P,MEI Q.Enquiring minds:Early detection of rumors in social media from enquiry posts[C]//Procee-dings of the 24th International Conference on World Wide Web.2015:1395-1405.
[8]TSCHIATSCHEK S,SINGLA A,GOMEZ R M,et al.FakeNews Detection in Social Networks via Crowd Signals[C]//Companion of The Web Conference 2018 on The Web Confe-rence 2018.2018:517-524.
[9]GUPTA A,LAMBA H,KUMARAGURU P,et al.Fakingsandy:characterizing and identifying fake images on twitter during hurricane sandy[C]//Proceedings of the 22nd International Conference on World Wide Web.ACM,2013:729-736.
[10]JIN Z,CAO J,ZHANG Y,et al.Novel visual and statistical ima-ge features for microblogs news verification[J].IEEE Transactions on Multimedia,2017,19(3):598-608.
[11]LIU Z,WEI Z H,ZHANG R X.Rumor detection based on con-volutional neural network[J].Journal of Computer Applications,2017(11):21-24,68.
[12]LIAO X W,HUANG Z,YANG D D,et al.Rumor detection in social media based on a hierarchical attention network[J].Scientia Sinica Informationis,2018,48(11):1558-1574.
[13]MA J,GAO W,WONG K.Rumor detection on twitter withtree-structured recursive neural networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.2018:1980-1989.
[14]RUCHANSKY N,SEO S,LIU Y.CSI:A hybrid deep model for fake news detection[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management.ACM,2017:797-806.
[15]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[16]ZHANG J,DONG B,YU P S.Deep Diffusive Neural Network based Fake News Detection from Heterogeneous Social Networks[C]//Proceedings of IEEE International Conference on Big Data.2019:1259-1266.
[17]SHU K,WANG S,LIU H.Beyond news contents:The role of social context for fake news detection[C]//Proceedings of the 12th ACM International Conference on Web Search and Data Mining.2019:312-320.
[18]LIU Y,WU Y.Early detection of fake news on social media through propagation path classification with recurrent andcon-volutional networks[C]//Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence.2018:354-361.
[19]MIKOLOV T,KARAFIAT M,BURGET L,et al.Recurrentneural network based language model[C]//Proceedings of 11th Annual Conference of the International Speech Communication Association.2010:1045-1048.
[20]YANG Y,ZHENG L,ZHANG J,et al.TI-CNN:Convolutional Neural Networks for Fake News Detection[J].arXiv:1806.00749,2018.
[21]JIN Z,CAO J,HAN G,et al.Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]//Proceedings of the 2017 ACM on Multimedia Conference.ACM,2017:795-816.
[22]WANG Y,MA F,JIN Z,et al.EANN:Event Adversarial Neural Networks for Multi-Modal Fake News Detection[C]//Procee-dings of The 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.ACM,2018:849-857.
[23]SCHUSTER M,PALIWAL K.Bidirectional recurrent neuralnetworks[J].IEEE Transactions on Signal Processing,1997,45(11):2673-2681.
[24]KIM Y.Convolutional Neural Networks for Sentence Classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP).ACM,2014:1746-1751.
[25]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Computer Society Confe-rence on Computer Vision and Pattern Recognition.ACM 2014:580-587.
[26]REN S,HE K,GIRSHICK R,et al.Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017(6):1137-1149.
[27]VASWANI A,SHAZEER N,PARMAR N,et al.Attention IsAll You Need[J].Advances in Neural Information Processing Systems,2017(12):5999-6009.
[28]GU Y,YANG K,FU S,et al.Hybrid Attention based Multimodal Network for Spoken Language Classification[C]//Procee-dings of the 27th International Conference on Computational Linguistics.ACM,2018:2379-2390.
[29]BOIDIDOU C,PAPADOPOULOS S,ZAMPOGLOU M,et al.Detection and visualization of misleading content on Twitter[J].International Journal of Multimedia Information Retrieval,2018,7(1):71-86.
[30]SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[C]//Proceedings of International Conference on Learning Representations.ICLR,2015:1-14.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 聂秀山, 潘嘉男, 谭智方, 刘新放, 郭杰, 尹义龙.
基于自然语言的视频片段定位综述
Overview of Natural Language Video Localization
计算机科学, 2022, 49(9): 111-122. https://doi.org/10.11896/jsjkx.220500130
[4] 周旭, 钱胜胜, 李章明, 方全, 徐常胜.
基于对偶变分多模态注意力网络的不完备社会事件分类方法
Dual Variational Multi-modal Attention Network for Incomplete Social Event Classification
计算机科学, 2022, 49(9): 132-138. https://doi.org/10.11896/jsjkx.220600022
[5] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[6] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[7] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[8] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[9] 刘冬梅, 徐洋, 吴泽彬, 刘倩, 宋斌, 韦志辉.
基于边框距离度量的增量目标检测方法
Incremental Object Detection Method Based on Border Distance Measurement
计算机科学, 2022, 49(8): 136-142. https://doi.org/10.11896/jsjkx.220100132
[10] 王灿, 刘永坚, 解庆, 马艳春.
基于软标签和样本权重优化的Anchor Free目标检测算法
Anchor Free Object Detection Algorithm Based on Soft Label and Sample Weight Optimization
计算机科学, 2022, 49(8): 157-164. https://doi.org/10.11896/jsjkx.210600240
[11] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[12] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[13] 程成, 降爱莲.
基于多路径特征提取的实时语义分割方法
Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction
计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[14] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[15] 周慧, 施皓晨, 屠要峰, 黄圣君.
基于主动采样的深度鲁棒神经网络学习
Robust Deep Neural Network Learning Based on Active Sampling
计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!