融合多模态信息的社交网络谣言检测方法

doi:10.11896/jsjkx.200400057

Abstract

Abstract: With the development of social networking platforms,social networks have become an important source of information for people.However,the convenience of social networks has also led to the rapid propagation of false rumors.Compared with textual rumors,social network rumors with multimedia content are more likely to mislead users and get dissemination,so the detection of multi-modal rumors is of great significance in real life.Several multi-modal rumor detection methods have been proposed,but the visual features and joint representation of text and visual features have not been fully explored in current approaches.To make up for these shortcomings,an end-to-end multi-modal fusion network based on deep learning is developed.Firstly,the visual features of each region of interest in the image are extracted.Then,the text and the visual features are updated and fused by using a multi-head attention mechanism.Finally,these features are concatenated based on the attention mechanism for the detection of multi-modal rumors in social networks.Comparative experiments on the public data sets of Twitter and Weibo are conducted and experimental results show that the proposed method has a 13.4% F1 value increase on Twitter data set and a 1.6% F1 value increase on Weibo data set.

Key words: Deep learning, Multi-moda, Object detection, Rumor detection

CLC Number:

TP391

ZHANG Shao-qin, DU Sheng-dong, ZHANG Xiao-bo, LI Tian-rui. Social Rumor Detection Method Based on Multimodal Fusion[J].Computer Science, 2021, 48(5): 117-123.

References

[1]CHEN Y F,LI Z Y,LIANG X,et al.Review on Rumor Detection of Online Social Networks[J].Chinese Journal of Compu-ters,2018,41(7):1648-1677.
[2]KARPATHY A,LI F F.Deep Visual-Semantic Alignments for Generating Image Descriptions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(4):664-676.
[3]ANDERSON P,HE X,BUEHLER C,et al.Bottom-Up andTop-Down Attention for Image Captioning and Visual Question Answering[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.ACM,2018:6077-6086.
[4]GAO P,JIANG Z,YOU H,et al.Dynamic fusion with intra-and inter-modality attention flow for visual question answering[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.ACM,2019:6632-6641.
[5]BALTRUSAITIS T,AHUJA C,MORENCY L P.Multimodal Machine Learning:A Survey and Taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(2):423-443.
[6]CASTILLO C,MENDOZA M,POBLETE B.Predicting infor-mation credibility in time-sensitive social media[J].Internet Research,2013,23(5):560-588.
[7]ZHAO Z,RESNICK P,MEI Q.Enquiring minds:Early detection of rumors in social media from enquiry posts[C]//Procee-dings of the 24th International Conference on World Wide Web.2015:1395-1405.
[8]TSCHIATSCHEK S,SINGLA A,GOMEZ R M,et al.FakeNews Detection in Social Networks via Crowd Signals[C]//Companion of The Web Conference 2018 on The Web Confe-rence 2018.2018:517-524.
[9]GUPTA A,LAMBA H,KUMARAGURU P,et al.Fakingsandy:characterizing and identifying fake images on twitter during hurricane sandy[C]//Proceedings of the 22nd International Conference on World Wide Web.ACM,2013:729-736.
[10]JIN Z,CAO J,ZHANG Y,et al.Novel visual and statistical ima-ge features for microblogs news verification[J].IEEE Transactions on Multimedia,2017,19(3):598-608.
[11]LIU Z,WEI Z H,ZHANG R X.Rumor detection based on con-volutional neural network[J].Journal of Computer Applications,2017(11):21-24,68.
[12]LIAO X W,HUANG Z,YANG D D,et al.Rumor detection in social media based on a hierarchical attention network[J].Scientia Sinica Informationis,2018,48(11):1558-1574.
[13]MA J,GAO W,WONG K.Rumor detection on twitter withtree-structured recursive neural networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.2018:1980-1989.
[14]RUCHANSKY N,SEO S,LIU Y.CSI:A hybrid deep model for fake news detection[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management.ACM,2017:797-806.
[15]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[16]ZHANG J,DONG B,YU P S.Deep Diffusive Neural Network based Fake News Detection from Heterogeneous Social Networks[C]//Proceedings of IEEE International Conference on Big Data.2019:1259-1266.
[17]SHU K,WANG S,LIU H.Beyond news contents:The role of social context for fake news detection[C]//Proceedings of the 12th ACM International Conference on Web Search and Data Mining.2019:312-320.
[18]LIU Y,WU Y.Early detection of fake news on social media through propagation path classification with recurrent andcon-volutional networks[C]//Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence.2018:354-361.
[19]MIKOLOV T,KARAFIAT M,BURGET L,et al.Recurrentneural network based language model[C]//Proceedings of 11th Annual Conference of the International Speech Communication Association.2010:1045-1048.
[20]YANG Y,ZHENG L,ZHANG J,et al.TI-CNN:Convolutional Neural Networks for Fake News Detection[J].arXiv:1806.00749,2018.
[21]JIN Z,CAO J,HAN G,et al.Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]//Proceedings of the 2017 ACM on Multimedia Conference.ACM,2017:795-816.
[22]WANG Y,MA F,JIN Z,et al.EANN:Event Adversarial Neural Networks for Multi-Modal Fake News Detection[C]//Procee-dings of The 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.ACM,2018:849-857.
[23]SCHUSTER M,PALIWAL K.Bidirectional recurrent neuralnetworks[J].IEEE Transactions on Signal Processing,1997,45(11):2673-2681.
[24]KIM Y.Convolutional Neural Networks for Sentence Classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP).ACM,2014:1746-1751.
[25]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Computer Society Confe-rence on Computer Vision and Pattern Recognition.ACM 2014:580-587.
[26]REN S,HE K,GIRSHICK R,et al.Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017(6):1137-1149.
[27]VASWANI A,SHAZEER N,PARMAR N,et al.Attention IsAll You Need[J].Advances in Neural Information Processing Systems,2017(12):5999-6009.
[28]GU Y,YANG K,FU S,et al.Hybrid Attention based Multimodal Network for Spoken Language Classification[C]//Procee-dings of the 27th International Conference on Computational Linguistics.ACM,2018:2379-2390.
[29]BOIDIDOU C,PAPADOPOULOS S,ZAMPOGLOU M,et al.Detection and visualization of misleading content on Twitter[J].International Journal of Multimedia Information Retrieval,2018,7(1):71-86.
[30]SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[C]//Proceedings of International Conference on Learning Representations.ICLR,2015:1-14.

Related Articles 15

[1]	RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2]	TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[3]	ZHOU Xu, QIAN Sheng-sheng, LI Zhang-ming, FANG Quan, XU Chang-sheng. Dual Variational Multi-modal Attention Network for Incomplete Social Event Classification [J]. Computer Science, 2022, 49(9): 132-138.
[4]	XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[5]	WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[6]	HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[7]	JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[8]	LIU Dong-mei, XU Yang, WU Ze-bin, LIU Qian, SONG Bin, WEI Zhi-hui. Incremental Object Detection Method Based on Border Distance Measurement [J]. Computer Science, 2022, 49(8): 136-142.
[9]	WANG Can, LIU Yong-jian, XIE Qing, MA Yan-chun. Anchor Free Object Detection Algorithm Based on Soft Label and Sample Weight Optimization [J]. Computer Science, 2022, 49(8): 157-164.
[10]	SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[11]	HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[12]	CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[13]	HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[14]	ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[15]	SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Social Rumor Detection Method Based on Multimodal Fusion

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0