动词幻觉：揭示和评估多模态大语言模型中的动词概念幻觉
Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models
AI总结：
本文首次揭示多模态大语言模型中的动词概念幻觉问题，并提出基于丰富动词知识的调优方法以有效缓解该问题。
摘要：
多模态大语言模型（MLLMs）最近引起了广泛关注，并在OCR、VQA、描述生成等任务中展现出卓越的能力。然而，幻觉仍然是一个持续的问题。尽管已经提出了许多方法来减轻幻觉，这些方法在取得显著改进的同时，主要集中在缓解与对象/名词相关的概念幻觉上。动词概念对于理解人类行为至关重要，但一直被忽视。在本文中，到目前为止，我们是首次从多个角度研究MLLMs中的动词幻觉现象。我们的发现表明，大多数最先进的MLLMs都受到严重的动词幻觉影响。为了评估现有缓解对象概念幻觉方法在动词幻觉上的有效性，我们评估了这些方法并发现它们并不能有效解决动词幻觉问题。为了解决这一问题，我们提出了一种基于丰富动词知识的调优方法来减轻动词幻觉。实验结果表明，我们的方法显著减少了与动词相关的幻觉。
英文摘要：
Multimodal Large Language Models (MLLMs) have garnered significant attention recently and demonstrate outstanding capabilities in various tasks such as OCR, VQA, captioning, $\textit{etc}$. However, hallucination remains a persistent issue. While numerous methods have been proposed to mitigate hallucinations, achieving notable improvements, these methods primarily focus on mitigating hallucinations about $\textbf{object/noun-related}$ concepts. Verb concepts, crucial for understanding human actions, have been largely overlooked. In this paper, to the best of our knowledge, we are the $\textbf{first}$ to investigate the $\textbf{verb hallucination}$ phenomenon of MLLMs from various perspectives. Our findings reveal that most state-of-the-art MLLMs suffer from severe verb hallucination. To assess the effectiveness of existing mitigation methods for object concept hallucination on verb hallucination, we evaluated these methods and found that they do not effectively address verb hallucination. To address this issue, we propose a novel rich verb knowledge-based tuning method to mitigate verb hallucination. The experiment results demonstrate that our method significantly reduces hallucinations related to verbs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
