通过大语言模型之间的多模态辩论实现可解释的有害模因检测
Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
AI总结：
本文提出通过大语言模型多模态辩论生成无害与有害立场的冲突解释，并微调小型语言模型作为裁判进行多模态融合，实现可解释且性能更优的有害模因检测。
摘要：
社交媒体时代充斥着互联网模因，因此需要清晰理解并有效识别其中的有害内容。由于模因中蕴含的隐含意义并未通过表层文本和图像明确传达，这一任务面临重大挑战。然而，现有的有害模因检测方法并未提供可读的解释来揭示这种隐含意义以支持其检测决策。在本文中，我们提出了一种可解释的有害模因检测方法，该方法通过对来自无害立场和有害立场的相互冲突的理据进行推理来实现。具体而言，受大语言模型（LLMs）在文本生成和推理方面强大能力的启发，我们首先引发LLMs之间的多模态辩论，以生成源自矛盾论点的解释。然后，我们提出微调一个小型语言模型作为辩论裁判来进行有害性推断，以促进有害性理据与模因内部固有模态信息之间的多模态融合。通过这种方式，我们的模型能够利用来自无害论点和有害论点的多模态解释，对复杂且隐含的有害指示模式进行辩证推理。在三个公开模因数据集上的大量实验表明，我们的有害模因检测方法取得了比最先进方法更好的性能，并且在解释模型预测的模因有害性方面展现出优越的能力。
英文摘要：
The age of social media is flooded with Internet memes, necessitating a clear grasp and effective identification of harmful ones. This task presents a significant challenge due to the implicit meaning embedded in memes, which is not explicitly conveyed through the surface text and image. However, existing harmful meme detection methods do not present readable explanations that unveil such implicit meaning to support their detection decisions. In this paper, we propose an explainable approach to detect harmful memes, achieved through reasoning over conflicting rationales from both harmless and harmful positions. Specifically, inspired by the powerful capacity of Large Language Models (LLMs) on text generation and reasoning, we first elicit multimodal debate between LLMs to generate the explanations derived from the contradictory arguments. Then we propose to fine-tune a small language model as the debate judge for harmfulness inference, to facilitate multimodal fusion between the harmfulness rationales and the intrinsic multimodal information within memes. In this way, our model is empowered to perform dialectical reasoning over intricate and implicit harm-indicative patterns, utilizing multimodal explanations originating from both harmless and harmful arguments. Extensive experiments on three public meme datasets demonstrate that our harmful meme detection approach achieves much better performance than state-of-the-art methods and exhibits a superior capacity for explaining the meme harmfulness of the model predictions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
