基于大模型与知识图谱增强的仇恨模因检测框架

doi:10.11896/jsjkx.260500030

摘要/Abstract

摘要： 互联网模因因具有高度隐喻性与多模态特征,已成为网络仇恨言论传播的新型隐蔽载体。现有的端到端检测方法缺乏深层隐喻解码能力,而基于大语言模型的生成式推理又容易脱离视觉事实产生“推理幻觉”,且面临高参数部署瓶颈。针对上述问题,提出一种基于大模型与知识图谱增强的仇恨模因检测框架。该框架实现了感知与推理的模态解耦:首先,引入视觉结构先验字典精准提取图像要素;其次,结合知识图谱增强构建深层多跳逻辑证据链,建立深层语义关联;随后,设计“起草-审计-精炼”的反思式推理机制,通过跨模态自我审计强制锚定客观事实,从底层机制上遏制幻觉;最后,提出渐进式多模态知识蒸馏策略,将复杂推理能力高效迁移至轻量级目标模型。实验验证,所提框架在Harm-P数据集上的检测准确率高达91.92%,优于现有基线方法。特别地,模型生成解释的事实一致性获4.63分(满分5分),显著抑制了推理幻觉;在跨架构实验中,该框架使基础模型准确率最高提升33.82个百分点。

关键词: 仇恨模因检测, 知识图谱增强, 可解释性, 幻觉抑制, 知识蒸馏, 多模态大模型

Abstract: Internet memes,characterized by high metaphoricity and multimodal features,have emerged as a new and covert me-dium for disseminating online hate speech.Current end-to-end detection methods lack the capability for deep metaphorical decoding,while generative reasoning based on large language models is prone to reasoning hallucinations detached from visual facts and faces bottlenecks in high-parameter deployment.To address these issues,this paper proposes a hateful meme detection framework enhanced by large models and knowledge graphs.The framework achieves the modal decoupling of perception and rea-soning.Firstly,a visual structure prior dictionary is introduced to accurately extract image elements.Secondly,deep multi-hop logical evidence chains are constructed through knowledge graph augmentation to establish profound semantic associations.Subsequently,a “draft-audit-refine” reflective reasoning mechanism is designed to anchor reasoning in objective facts through cross-modal self-audit,curbing hallucinations at the algorithmic level.Finally,a progressive multimodal knowledge distillation strategy is proposed to efficiently transfer complex reasoning capabilities to lightweight target models.Experimental results demonstrate that the proposed framework achieves a detection accuracy of 91.92% on the Harm-P dataset,outperforming existing baseline methods.Notably,the faithfulness score of the generated explanations reaches 4.63(out of 5),significantly suppressing reasoning hallucinations.In cross-architecture experiments,the framework improves the accuracy of base models by up to 33.82 percentage points.

Key words: Hateful meme detection, Knowledge graph augmentation, Interpretability, Hallucination suppression, Knowledge distillation, Multimodal large language models

中图分类号:

TP391

成城, 钟寒. 基于大模型与知识图谱增强的仇恨模因检测框架[J]. 计算机科学, 2026, 53(10): 77-91. https://doi.org/10.11896/jsjkx.260500030

CHENG Cheng, ZHONG Han. Hateful Meme Detection Framework Enhanced by Large Models and Knowledge Graphs[J]. Computer Science, 2026, 53(10): 77-91. https://doi.org/10.11896/jsjkx.260500030

参考文献

[1] INSIGHTS S.Global Social Media Statistics Research Summary 2025 [EB/OL].(2025-01-15) [2026-05-20].https://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-social-media-research/.
[2] GANGULY A,BIN EMRAN A N,PUSPO S S C,et al.Masonperplexity at Multimodal Hate Speech Event Detection 2024:Hate Speech and Target Detection Using Transformer Ensembles [C]//Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text(CASE 2024).Stroudsburg:Association for Computational Linguistics,2024:125-131.
[3] MADRIAZA P,HASSAN G,BROUILLETTE-ALARIE S,et al.Exposure to Hate in Online and Traditional Media:A Systematic Review and Meta-Analysis of the Impact of This Exposure on Individuals and Communities [J].Campbell Systematic Reviews,2025,21(1):e70018.
[4] AMIN F,HORAIRA M A,SHAWON M T A,et al.Phan-tomTroupe@ CASE 2025:Multimodal Hate Speech Detection in Text-Embedded Memes using Instruction-Tuned LLMs [C]//Proceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Texts.Stroudsburg:Association for Computational Linguistics,2025:133-138.
[5] SHARMA S,ALAM F,AKHTAR M S,et al.Detecting andUnderstanding Harmful Memes:A Survey [J].arXiv:2205.04274,2022.
[6] DANILEVSKY M,QIAN K,AHARONOV R,et al.A Survey of the State of Explainable AI for Natural Language Processing [C]//Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2020:447-459.
[7] YIN S,FU C,ZHAO S,et al.A Survey on Multimodal LargeLanguage Models [J].National Science Review,2024,11(12):nwae403.
[8] LEWIS P,PEREZ E,PIKTUS A,et al.Retrieval-AugmentedGeneration for Knowledge-Intensive NLP Tasks [J].Advances in Neural Information Processing Systems,2020,33:9459-9474.
[9] ZHANG Z,ZHANG A,LI M,et al.Multimodal Chain-of-Thought Reasoning in Language Models [J].arXiv:2302.00923,2023.
[10] PAN S,LUO L,WANG Y,et al.Unifying Large LanguageModels and Knowledge Graphs:A Roadmap [J].IEEE Transactions on Knowledge and Data Engineering,2024,36(7):3580-3599.
[11] LI Y,DU Y,ZHOU K,et al.Evaluating Object Hallucination in Large Vision-Language Models [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proces-sing.Stroudsburg:Association for Computational Linguistics,2023:292-305.
[12] HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:770-778.
[13] DEVLIN J,CHANG M,LEE K,et al.BERT:Pre-training ofDeep Bidirectional Transformers for Language Understanding [C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Stroudsburg:Association for Computational Linguistics,2019:4171-4186.
[14] PRAMANICK S,SHARMA S,DIMITROV D,et al.MOMENTA:A Multimodal Framework for Detecting Harmful Memes and Their Targets [C]//Findings of the Association for Computational Linguistics:EMNLP 2021.Stroudsburg:Association for Computational Linguistics,2021:4439-4455.
[15] LEE R K,CAO R,FAN Z,et al.Disentangling Hate in Online Memes [C]//Proceedings of the 29th ACM International Conference on Multimedia.New York:ACM Press,2021:5138-5147.
[16] PONNUSAMY R,RAJIAKODI S,SIVAGNANAM B,et al.Fuse-MD:A Culturally-Aware Multimodal Model for Detecting Misogyny Memes [J].Natural Language Processing Journal,2026,8:100197.
[17] HUANG J,PAN J,WAN Z,et al.Evolver:Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection [C]//Proceedings of the 31st International Confe-rence on Computational Linguistics.Stroudsburg:Association for Computational Linguistics,2025:7321-7330.
[18] LIN H,LUO Z,GAO W,et al.Towards Explainable Harmful Meme Detection Through Multimodal Debate Between Large Language Models [C]//Proceedings of the ACM Web Confe-rence 2024.New York:ACM Press,2024:2359-2370.
[19] ZHUANG Y,GUO K,WANG J,et al.I Know What YouMeme!Understanding and Detecting Harmful Memes with Multimodal Large Language Models [J].arXiv:2502.03456,2025.
[20] MEI J,SUN M,CHEN J,et al.ExPO-HM:Learning to Explain-then-Detect for Hateful Meme Detection [J].arXiv:2510.08630,2025.
[21] MEI J,CHEN J,YANG G,et al.Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2025:23806-23828.
[22] TZELEPI M,MEZARIS V.Improving Multimodal HatefulMeme Detection Exploiting LMM-Generated Knowledge [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop.New York:IEEE Press,2025:202-211.
[23] KMAINASI M B,HASNAT A,HASAN M A,et al.MemeIntel:Explainable Detection of Propagandistic and Hateful Memes [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2025:30263-30279.
[24] LIU O S,NG P C,SOH D W,et al.Yes FLoReNce,I Will Do Better Next Time! Agentic Feedback Reasoning for Humorous Meme Detection [J].arXiv:2601.07232,2026.
[25] CORMEN T H,LEISERSON C E,RIVEST R L,et al.Introduction to Algorithms [M].Cambridge:MIT Press,2022:530-560.
[26] HU E J,SHEN Y,WALLIS P,et al.LoRA:Low-Rank Adaptation of Large Language Models [J].International Conference on Learning Representations(ICLR),2022,1(2):3-11.
[27] PRAMANICK S,DIMITROV D,MUKHERJEE R,et al.Detecting Harmful Memes and Their Targets [C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing:Findings.Stroudsburg:Association for Computational Linguistics,2021:2788-2796.
[28] SHAH S B,SHIWAKOTI S,CHAUDHARY M,et al.MemeCLIP:Leveraging CLIP Representations for Multimodal Meme Classification [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2024:17320-17332.
[29] FERSINI E,GASPARINI F,RIZZI G,et al.SemEval-2022 Task 5:Multimedia Automatic Misogyny Identification [C]//Proceedings of the 16th International Workshop on Semantic Evaluation(SemEval-2022).Stroudsburg:Association for Computational Linguistics,2022:533-541.
[30] ZHENG L,CHIANG W,SHENG Y,et al.Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [J].Advances in Neural Information Processing Systems,2023,36:46595-46623.
[31] LIU A,FENG B,WANG B,et al.DeepSeek-V2:A Strong,Economical,and Efficient Mixture-of-Experts Language Model [J].arXiv:2405.04434,2024.
[32] MATHEW B,SAHA P,YIMAM S M,et al.HateXplain:ABenchmark Dataset for Explainable Hate Speech Detection [C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2021:14867-14875.
[33] LIU Y,ITER D,XU Y,et al.G-Eval:NLG Evaluation using GPT-4 with Better Human Alignment [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2023:2510-2522.
[34] VAN DER LEE C,GATT A,VAN MILTENBURG E,et al.Best Practices for the Human Evaluation of Automatically Ge-nerated Text [C]//Proceedings of the 12th International Confe-rence on Natural Language Generation.Stroudsburg:Association for Computational Linguistics,2019:355-368.
[35] BAI S,CHEN K,LIU X,et al.Qwen2.5-VL Technical Report [J].arXiv:2502.10001,2025.
[36] CHUNG H W,HOU L,LONGPRE S,et al.Scaling Instruction-Finetuned Language Models [J].Journal of Machine Learning Research,2024,25(70):1-53.
[37] ZHENG Y,ZHANG R,ZHANG J,et al.LlamaFactory:Unified Efficient Fine-Tuning of 100+ Language Models [C]//Procee-dings of the 62nd Annual Meeting of the Association for Computational Linguistics:System Demonstrations.Stroudsburg:Association for Computational Linguistics,2024:400-410.
[38] LOSHCHILOV I,HUTTER F.Decoupled Weight Decay Regularization [J].arXiv:1711.05101,2017.
[39] ANDERSON P,HE X,BUEHLER C,et al.Bottom-Up andTop-Down Attention for Image Captioning and Visual Question Answering [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:6077-6086.
[40] KIELA D,FIROOZ H,MOHAN A,et al.The Hateful Memes Challenge:Detecting Hate Speech in Multimodal Memes [J].Advances in Neural Information Processing Systems,2020,33:2611-2624.
[41] KIELA D,BHOOSHAN S,FIROOZ H,et al.Supervised Multimodal Bitransformers for Classifying Images and Text [J].ar-Xiv:1909.02950,2019.
[42] LI L H,YATSKAR M,YIN D,et al.VisualBERT:A Simple and Performant Baseline for Vision and Language [J].arXiv:1908.03557,2019.
[43] LU J,BATRA D,PARIKH D,et al.ViLBERT:PretrainingTask-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks [J].Advances in Neural Information Proces-sing Systems,2019,32:13-23.
[44] CAO R,LEE R K,CHONG W,et al.Prompting for Multimodal Hateful Meme Classification [C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:Association for Computational Linguistics,2022:321-332.
[45] CAO R,HEE M S,KUEK A,et al.Pro-Cap:Leveraging a Frozen Vision-Language Model for Hateful Meme Detection [C]//Proceedings of the 31st ACM International Conference on Multimedia.New York:ACM Press,2023:5244-5252.
[46] ZHANG B,ZHANG P,DONG X,et al.Long-CLIP:Unlocking the Long-Text Capability of CLIP [C]//Proceedings of the European Conference on Computer Vision.Berlin:Springer,2024:310-325.
[47] LIU H,LI C,LI Y,et al.Improved Baselines with Visual Instruction Tuning [C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2024:26296-26306.
[48] GRATTAFIORI A,DUBEY A,JAUHRI A,et al.The Llama 3Herd of Models [J].arXiv:2407.21783,2024.
[49] MADAAN A,TANDON N,GUPTA P,et al.Self-refine:Iterative refinement with self-feedback [J].Advances in Neural Information Processing Systems,2023,36:46534-46594.
[50] BESTA M,BLACH N,KUBICEK A,et al.Graph of thoughts:Solving elaborate problems with large language models [C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2024:17682-17690.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed