LEMMA：面向LVLM增强的多模态虚假信息检测与外部知识增强
LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation
AI总结：
针对多模态虚假信息检测难题，本文提出融合外部知识增强的LVLM方法LEMMA，在两个数据集上较基线分别提升7%和13%的准确率。
摘要：
社交平台上多模态虚假信息的兴起给个人和社会带来了严峻挑战。与文本虚假信息相比，其可信度更高、影响范围更广，这使得检测工作十分复杂，需要具备跨不同媒体类型的稳健推理能力以及深厚的知识储备才能实现准确验证。大视觉语言模型（Large Vision Language Model，LVLM）的出现为该问题提供了潜在解决方案。凭借其在处理视觉和文本信息方面的优势，LVLM在识别复杂信息方面展现出可观的能力，并具备强大的推理技能。本文首先探究了LVLM在多模态虚假信息检测方面的潜力。我们发现，尽管LVLM的性能优于大语言模型（LLMs），但在缺乏证据的情况下，其深度推理能力可能会表现有限。基于这些观察，我们提出了LEMMA：带外部知识增强的LVLM增强多模态虚假信息检测方法。LEMMA利用LVLM的直觉和推理能力，同时通过外部知识对其进行增强，以提升虚假信息检测的准确率。我们的方法在Twitter和Fakeddit数据集上的准确率分别比顶级基线LVLM提升了7%和13%。
英文摘要：
The rise of multimodal misinformation on social platforms poses significant challenges for individuals and societies. Its increased credibility and broader impact compared to textual misinformation make detection complex, requiring robust reasoning across diverse media types and profound knowledge for accurate verification. The emergence of Large Vision Language Model (LVLM) offers a potential solution to this problem. Leveraging their proficiency in processing visual and textual information, LVLM demonstrates promising capabilities in recognizing complex information and exhibiting strong reasoning skills. In this paper, we first investigate the potential of LVLM on multimodal misinformation detection. We find that even though LVLM has a superior performance compared to LLMs, its profound reasoning may present limited power with a lack of evidence. Based on these observations, we propose LEMMA: LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation. LEMMA leverages LVLM intuition and reasoning capabilities while augmenting them with external knowledge to enhance the accuracy of misinformation detection. Our method improves the accuracy over the top baseline LVLM by 7% and 13% on Twitter and Fakeddit datasets respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
