通过自增强对比对齐缓解多模态大语言模型中的对象和动作幻觉
Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment
AI总结：
SANTA框架通过自增强对比对齐方法，有效缓解多模态大语言模型中的对象和动作幻觉问题。
摘要：
近年来，多模态大语言模型（MLLMs）在生成输入视频描述性标题方面展示了显著的能力。然而，这些模型在生成的描述中存在事实不准确的问题，导致严重的幻觉问题。尽管先前的工作已经探索了减轻静态图像中的幻觉，但联合缓解动态视频中的视觉对象和时间动作幻觉仍然是一个具有挑战性和未解决的任务。为了解决这一挑战，我们提出了一种自增强对比对齐（SANTA）框架，以通过排除虚假相关性并强调视觉事实来实现对象和动作的忠实性。SANTA采用了一种幻觉自增强方案，以识别MLLM中潜在的幻觉，并将原始描述转换为对比负样本。此外，我们开发了一种轨迹-短语对比对齐，以将区域对象和关系引导的动作与对应的视觉和时间短语匹配。广泛的实验表明，SANTA在缓解对象和动作幻觉方面优于现有方法，在幻觉检验基准上表现更优。
英文摘要：
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
