迈向鲁棒的超详细图像描述：一种多智能体方法及用于事实性与覆盖度的双重评估指标
Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
AI总结：
本文针对多模态大语言模型在超详细图像描述中的幻觉问题，提出LLM-MLLM协作的多智能体纠正方法，并构建双重评估框架与基准数据集，显著提升描述的事实准确性，同时揭示VQA基准与详细描述能力不相关的局限性。
摘要：
多模态大语言模型（MLLMs）擅长生成高度详细的描述，但经常产生幻觉。我们的分析揭示，现有的幻觉检测方法在处理详细描述时存在困难。我们将此归因于随着序列长度增长，MLLMs越来越依赖其生成的文本而非输入图像。为解决这一问题，我们提出了一种多智能体方法，利用LLM-MLLM协作来纠正给定的描述。此外，我们引入了一个评估框架和一个基准数据集，以促进对详细描述的系统性分析。我们的实验表明，我们提出的评估方法在事实性方面比现有指标更符合人类判断，并且现有的提高MLLM事实性的方法在超详细图像描述任务中可能力不从心。相比之下，我们提出的方法显著增强了描述的事实准确性，甚至改进了由GPT-4V生成的描述。最后，我们通过证明MLLM在VQA基准上的表现可能与其生成详细图像描述的能力不相关，揭示了以VQA为中心的基准评估的局限性。我们的代码和数据可在 https://github.com/adobe-research/CapMAS 获取。
英文摘要：
Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions. Our code and data are available at https://github.com/adobe-research/CapMAS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
