鲁棒适应大规模多模态模型用于检索增强的仇恨表情包检测
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
AI总结：
本文提出了一种鲁棒适应框架，用于提升大规模多模态模型在检索增强的仇恨表情包检测中的性能与泛化能力，同时提高模型的可解释性。
摘要：
仇恨表情包已成为互联网上的重大问题，需要鲁棒的自动检测系统。尽管大规模多模态模型（LMMs）在仇恨表情包检测中展现出潜力，但它们面临显著挑战，如性能不优和有限的领域外泛化能力。最近的研究进一步揭示了在这一设置中，监督微调（SFT）和上下文学习的局限性。为了解决这些问题，我们提出了一种鲁棒的适应框架，用于仇恨表情包检测，以提高领域内准确性与跨领域泛化能力，同时保持LMMs的一般视觉语言能力。分析表明，我们的方法在对抗攻击下比SFT模型更具鲁棒性。在六个表情包分类数据集上的实验显示，我们的方法实现了最先进的性能，优于更大的代理系统。此外，我们的方法生成的解释性理由比标准SFT更高质量，增强了模型的可解释性。代码可在https://github.com/JingbiaoMei/RGCL获取。
英文摘要：
Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While Large Multimodal Models (LMMs) have shown promise in hateful meme detection, they face notable challenges like sub-optimal performance and limited out-of-domain generalization capabilities. Recent studies further reveal the limitations of both supervised fine-tuning (SFT) and in-context learning when applied to LMMs in this setting. To address these issues, we propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. Analysis reveals that our approach achieves improved robustness under adversarial attacks compared to SFT models. Experiments on six meme classification datasets show that our approach achieves state-of-the-art performance, outperforming larger agentic systems. Moreover, our method generates higher-quality rationales for explaining hateful content compared to standard SFT, enhancing model interpretability. Code available at https://github.com/JingbiaoMei/RGCL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
