Computer Science ›› 2026, Vol. 53 ›› Issue (10): 77-91.doi: 10.11896/jsjkx.260500030

• Large Models and Multimodal Intelligent Perception • Previous Articles     Next Articles

Hateful Meme Detection Framework Enhanced by Large Models and Knowledge Graphs

CHENG Cheng, ZHONG Han   

  1. College of Information and Network Security,People’s Public Security University of China,Beijing 100038, China
  • Received:2026-05-11 Revised:2026-07-27 Published:2026-10-13
  • About author:CHENG Cheng,born in 2001,postgra-duate.His main research interests include hateful meme detection and artificial intelligence security.
    ZHONG Han,born in 1984,Ph.D,associate professor.Her main research interests include artificial intelligence security,network control technology and computer vision.
  • Supported by:
    Project of Introducing Talents for Discipline Innovation Base in Higher Education Institutions(B20087).

Abstract: Internet memes,characterized by high metaphoricity and multimodal features,have emerged as a new and covert me-dium for disseminating online hate speech.Current end-to-end detection methods lack the capability for deep metaphorical decoding,while generative reasoning based on large language models is prone to reasoning hallucinations detached from visual facts and faces bottlenecks in high-parameter deployment.To address these issues,this paper proposes a hateful meme detection framework enhanced by large models and knowledge graphs.The framework achieves the modal decoupling of perception and rea-soning.Firstly,a visual structure prior dictionary is introduced to accurately extract image elements.Secondly,deep multi-hop logical evidence chains are constructed through knowledge graph augmentation to establish profound semantic associations.Subsequently,a “draft-audit-refine” reflective reasoning mechanism is designed to anchor reasoning in objective facts through cross-modal self-audit,curbing hallucinations at the algorithmic level.Finally,a progressive multimodal knowledge distillation strategy is proposed to efficiently transfer complex reasoning capabilities to lightweight target models.Experimental results demonstrate that the proposed framework achieves a detection accuracy of 91.92% on the Harm-P dataset,outperforming existing baseline methods.Notably,the faithfulness score of the generated explanations reaches 4.63(out of 5),significantly suppressing reasoning hallucinations.In cross-architecture experiments,the framework improves the accuracy of base models by up to 33.82 percentage points.

Key words: Hateful meme detection, Knowledge graph augmentation, Interpretability, Hallucination suppression, Knowledge distillation, Multimodal large language models

CLC Number: 

  • TP391
[1] INSIGHTS S.Global Social Media Statistics Research Summary 2025 [EB/OL].(2025-01-15) [2026-05-20].https://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-social-media-research/.
[2] GANGULY A,BIN EMRAN A N,PUSPO S S C,et al.Masonperplexity at Multimodal Hate Speech Event Detection 2024:Hate Speech and Target Detection Using Transformer Ensembles [C]//Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text(CASE 2024).Stroudsburg:Association for Computational Linguistics,2024:125-131.
[3] MADRIAZA P,HASSAN G,BROUILLETTE-ALARIE S,et al.Exposure to Hate in Online and Traditional Media:A Systematic Review and Meta-Analysis of the Impact of This Exposure on Individuals and Communities [J].Campbell Systematic Reviews,2025,21(1):e70018.
[4] AMIN F,HORAIRA M A,SHAWON M T A,et al.Phan-tomTroupe@ CASE 2025:Multimodal Hate Speech Detection in Text-Embedded Memes using Instruction-Tuned LLMs [C]//Proceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Texts.Stroudsburg:Association for Computational Linguistics,2025:133-138.
[5] SHARMA S,ALAM F,AKHTAR M S,et al.Detecting andUnderstanding Harmful Memes:A Survey [J].arXiv:2205.04274,2022.
[6] DANILEVSKY M,QIAN K,AHARONOV R,et al.A Survey of the State of Explainable AI for Natural Language Processing [C]//Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2020:447-459.
[7] YIN S,FU C,ZHAO S,et al.A Survey on Multimodal LargeLanguage Models [J].National Science Review,2024,11(12):nwae403.
[8] LEWIS P,PEREZ E,PIKTUS A,et al.Retrieval-AugmentedGeneration for Knowledge-Intensive NLP Tasks [J].Advances in Neural Information Processing Systems,2020,33:9459-9474.
[9] ZHANG Z,ZHANG A,LI M,et al.Multimodal Chain-of-Thought Reasoning in Language Models [J].arXiv:2302.00923,2023.
[10] PAN S,LUO L,WANG Y,et al.Unifying Large LanguageModels and Knowledge Graphs:A Roadmap [J].IEEE Transactions on Knowledge and Data Engineering,2024,36(7):3580-3599.
[11] LI Y,DU Y,ZHOU K,et al.Evaluating Object Hallucination in Large Vision-Language Models [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proces-sing.Stroudsburg:Association for Computational Linguistics,2023:292-305.
[12] HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:770-778.
[13] DEVLIN J,CHANG M,LEE K,et al.BERT:Pre-training ofDeep Bidirectional Transformers for Language Understanding [C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Stroudsburg:Association for Computational Linguistics,2019:4171-4186.
[14] PRAMANICK S,SHARMA S,DIMITROV D,et al.MOMENTA:A Multimodal Framework for Detecting Harmful Memes and Their Targets [C]//Findings of the Association for Computational Linguistics:EMNLP 2021.Stroudsburg:Association for Computational Linguistics,2021:4439-4455.
[15] LEE R K,CAO R,FAN Z,et al.Disentangling Hate in Online Memes [C]//Proceedings of the 29th ACM International Conference on Multimedia.New York:ACM Press,2021:5138-5147.
[16] PONNUSAMY R,RAJIAKODI S,SIVAGNANAM B,et al.Fuse-MD:A Culturally-Aware Multimodal Model for Detecting Misogyny Memes [J].Natural Language Processing Journal,2026,8:100197.
[17] HUANG J,PAN J,WAN Z,et al.Evolver:Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection [C]//Proceedings of the 31st International Confe-rence on Computational Linguistics.Stroudsburg:Association for Computational Linguistics,2025:7321-7330.
[18] LIN H,LUO Z,GAO W,et al.Towards Explainable Harmful Meme Detection Through Multimodal Debate Between Large Language Models [C]//Proceedings of the ACM Web Confe-rence 2024.New York:ACM Press,2024:2359-2370.
[19] ZHUANG Y,GUO K,WANG J,et al.I Know What YouMeme!Understanding and Detecting Harmful Memes with Multimodal Large Language Models [J].arXiv:2502.03456,2025.
[20] MEI J,SUN M,CHEN J,et al.ExPO-HM:Learning to Explain-then-Detect for Hateful Meme Detection [J].arXiv:2510.08630,2025.
[21] MEI J,CHEN J,YANG G,et al.Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2025:23806-23828.
[22] TZELEPI M,MEZARIS V.Improving Multimodal HatefulMeme Detection Exploiting LMM-Generated Knowledge [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop.New York:IEEE Press,2025:202-211.
[23] KMAINASI M B,HASNAT A,HASAN M A,et al.MemeIntel:Explainable Detection of Propagandistic and Hateful Memes [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2025:30263-30279.
[24] LIU O S,NG P C,SOH D W,et al.Yes FLoReNce,I Will Do Better Next Time! Agentic Feedback Reasoning for Humorous Meme Detection [J].arXiv:2601.07232,2026.
[25] CORMEN T H,LEISERSON C E,RIVEST R L,et al.Introduction to Algorithms [M].Cambridge:MIT Press,2022:530-560.
[26] HU E J,SHEN Y,WALLIS P,et al.LoRA:Low-Rank Adaptation of Large Language Models [J].International Conference on Learning Representations(ICLR),2022,1(2):3-11.
[27] PRAMANICK S,DIMITROV D,MUKHERJEE R,et al.Detecting Harmful Memes and Their Targets [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing:Findings.Stroudsburg:Association for Computational Linguistics,2021:2788-2796.
[28] SHAH S B,SHIWAKOTI S,CHAUDHARY M,et al.MemeCLIP:Leveraging CLIP Representations for Multimodal Meme Classification [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2024:17320-17332.
[29] FERSINI E,GASPARINI F,RIZZI G,et al.SemEval-2022 Task 5:Multimedia Automatic Misogyny Identification [C]//Proceedings of the 16th International Workshop on Semantic Evaluation(SemEval-2022).Stroudsburg:Association for Computational Linguistics,2022:533-541.
[30] ZHENG L,CHIANG W,SHENG Y,et al.Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [J].Advances in Neural Information Processing Systems,2023,36:46595-46623.
[31] LIU A,FENG B,WANG B,et al.DeepSeek-V2:A Strong,Economical,and Efficient Mixture-of-Experts Language Model [J].arXiv:2405.04434,2024.
[32] MATHEW B,SAHA P,YIMAM S M,et al.HateXplain:ABenchmark Dataset for Explainable Hate Speech Detection [C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2021:14867-14875.
[33] LIU Y,ITER D,XU Y,et al.G-Eval:NLG Evaluation using GPT-4 with Better Human Alignment [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2023:2510-2522.
[34] VAN DER LEE C,GATT A,VAN MILTENBURG E,et al.Best Practices for the Human Evaluation of Automatically Ge-nerated Text [C]//Proceedings of the 12th International Confe-rence on Natural Language Generation.Stroudsburg:Association for Computational Linguistics,2019:355-368.
[35] BAI S,CHEN K,LIU X,et al.Qwen2.5-VL Technical Report [J].arXiv:2502.10001,2025.
[36] CHUNG H W,HOU L,LONGPRE S,et al.Scaling Instruction-Finetuned Language Models [J].Journal of Machine Learning Research,2024,25(70):1-53.
[37] ZHENG Y,ZHANG R,ZHANG J,et al.LlamaFactory:Unified Efficient Fine-Tuning of 100+ Language Models [C]//Procee-dings of the 62nd Annual Meeting of the Association for Computational Linguistics:System Demonstrations.Stroudsburg:Association for Computational Linguistics,2024:400-410.
[38] LOSHCHILOV I,HUTTER F.Decoupled Weight Decay Regularization [J].arXiv:1711.05101,2017.
[39] ANDERSON P,HE X,BUEHLER C,et al.Bottom-Up andTop-Down Attention for Image Captioning and Visual Question Answering [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:6077-6086.
[40] KIELA D,FIROOZ H,MOHAN A,et al.The Hateful Memes Challenge:Detecting Hate Speech in Multimodal Memes [J].Advances in Neural Information Processing Systems,2020,33:2611-2624.
[41] KIELA D,BHOOSHAN S,FIROOZ H,et al.Supervised Multimodal Bitransformers for Classifying Images and Text [J].ar-Xiv:1909.02950,2019.
[42] LI L H,YATSKAR M,YIN D,et al.VisualBERT:A Simple and Performant Baseline for Vision and Language [J].arXiv:1908.03557,2019.
[43] LU J,BATRA D,PARIKH D,et al.ViLBERT:PretrainingTask-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks [J].Advances in Neural Information Proces-sing Systems,2019,32:13-23.
[44] CAO R,LEE R K,CHONG W,et al.Prompting for Multimodal Hateful Meme Classification [C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2022:321-332.
[45] CAO R,HEE M S,KUEK A,et al.Pro-Cap:Leveraging a Frozen Vision-Language Model for Hateful Meme Detection [C]//Proceedings of the 31st ACM International Conference on Multimedia.New York:ACM Press,2023:5244-5252.
[46] ZHANG B,ZHANG P,DONG X,et al.Long-CLIP:Unlocking the Long-Text Capability of CLIP [C]//Proceedings of the European Conference on Computer Vision.Berlin:Springer,2024:310-325.
[47] LIU H,LI C,LI Y,et al.Improved Baselines with Visual Instruction Tuning [C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2024:26296-26306.
[48] GRATTAFIORI A,DUBEY A,JAUHRI A,et al.The Llama 3Herd of Models [J].arXiv:2407.21783,2024.
[49] MADAAN A,TANDON N,GUPTA P,et al.Self-refine:Iterative refinement with self-feedback [J].Advances in Neural Information Processing Systems,2023,36:46534-46594.
[50] BESTA M,BLACH N,KUBICEK A,et al.Graph of thoughts:Solving elaborate problems with large language models [C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2024:17682-17690.
[1] HU Yihui, LI Yan, GUO Tianxiang, LI Wentao, HU Xingchen. Ship Recognition Methods via Rule Construction from Distributed Incomplete Data [J]. Computer Science, 2026, 53(9): 136-144.
[2] ZHAO Junlong, XU Bin, XIN Weiwei. Adversarial Knowledge Distillation Framework Integrating Temperature Removal Mechanism and Mutual Information Enhancement Strategy [J]. Computer Science, 2026, 53(9): 283-290.
[3] ZHONG Rui, YAN Hongwei, LIU Jiawei. Lightweight Low-resolution Face Recognition via Hierarchical Dynamic Feature Generation Distillation [J]. Computer Science, 2026, 53(8): 209-218.
[4] PING Fengqin, FU Xiaodong. Personalized Federated Learning for Concept and Label Distribution Drift [J]. Computer Science, 2026, 53(7): 433-441.
[5] LI Siyu, QIAN Wenhua. HCKD:Lightweight Skin Lesion Classification Method Based on Dermoscopic Images [J]. Computer Science, 2026, 53(6A): 250600143-9.
[6] FU Shiqi, ZHU Jinxia, XU Qichen, DU Zeyu. Dynamic Sparsity and Heterogeneous Knowledge Distillation for Top-k Recommendation [J]. Computer Science, 2026, 53(6A): 250700121-9.
[7] SONG Jianhua, LIU Chun, ZHANG Yan. Lightweight Camouflaged Object Detection Model Based on Structured Knowledge Distillation [J]. Computer Science, 2026, 53(4): 299-307.
[8] LI Yidan, CUI Jianying, XIONG Minghui. Category-Theoretic Semantic Representation: Systematic Review and Compositional Mechanism Analysis [J]. Computer Science, 2026, 53(4): 337-346.
[9] PAN Jiahao, FENG Xiang, YU Huiqun. SM-PHT:Robust,Scalable,and Efficient Method for Multi-task Reinforcement Learning [J]. Computer Science, 2026, 53(4): 366-376.
[10] WU Qiaorui, LUO Li, ZHAO Cairong. LLM-augmented Training Framework with Cycle-Consistency Constraints [J]. Computer Science, 2026, 53(4): 377-383.
[11] LIU Zhenyu, YAO Dengfeng, ZHAO Yuan, MENG Ziwen, XU Cheng, LI Yan. Research Progress of Multimodal Machine Learning in Audio-Visual Semantic Analysis [J]. Computer Science, 2026, 53(10): 1-17.
[12] WEN Zerui, JIANG Tian, HUANG Zijian, CUI Xiaohui. Section Sparse Attack:A More Powerful Sparse Attack Method [J]. Computer Science, 2026, 53(1): 323-330.
[13] JIANG Yunliang, JIN Senyang, ZHANG Xiongtao, LIU Kaining, SHEN Qing. Multi-scale Multi-granularity Decoupled Distillation Fuzzy Classifier and Its Application inEpileptic EEG Signal Detection [J]. Computer Science, 2025, 52(9): 37-46.
[14] DENG Jiayan, TIAN Shirui, LIU Xiangli, OUYANG Hongwei, JIAO Yunjia, DUAN Mingxing. Trajectory Prediction Method Based on Multi-stage Pedestrian Feature Mining [J]. Computer Science, 2025, 52(9): 241-248.
[15] LIU Le, XIAO Rong, YANG Xiao. Application of Decoupled Knowledge Distillation Method in Document-level RelationExtraction [J]. Computer Science, 2025, 52(8): 277-287.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!