基于LVLM的图像字幕生成中，更多细节是否总会引入更多幻觉？
Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?
AI总结：
针对“更多细节是否总会让LVLM图像字幕生成产生更多幻觉”的争议，提出差异化波束解码策略DBD与CLIP系列评估指标，在Visual Genome数据集上验证了其可生成细节丰富且低幻觉的字幕。
摘要：
大视觉语言模型（LVLM）擅长整合视觉与语言语境以生成细节丰富的内容，为图像字幕生成等应用提供了便利。然而，使用LVLM生成描述时常面临对象幻觉（OH）问题，即输出文本会错误呈现输入图像中的实际对象。尽管此前研究将OH的产生归因于包含更多细节，但本研究发现现有指标存在技术缺陷，导致对模型的评估及关于OH的结论并不可靠，这也引发了关于“基于LVLM的图像字幕生成中，更多细节是否总会引入更多幻觉？”这一问题的争论。\n 在本文中，我们为解决这一争论提出了一种新型解码策略——差异化波束解码（DBD），以及一套可靠的新型评估指标：CLIP-Precision、CLIP-Recall和CLIP-F1。DBD将视觉输入中隐藏的丰富信息并行解码为被称为“单元事实”的不同语言表示，该解码通过精心设计的差异化分数实现，该分数可引导并行搜索与候选筛选，随后将筛选出的单元事实聚合以生成最终字幕。我们提出的指标通过对比真实图像区域与生成文本分区的嵌入组，来评估图像字幕的全面性与准确性。在Visual Genome数据集上开展的大量实验验证了我们方法的有效性，结果表明该方法既能生成细节丰富的描述，又能保持较低的幻觉水平。
英文摘要：
Large Vision-Language Models (LVLMs) excel in integrating visual and linguistic contexts to produce detailed content, facilitating applications such as image captioning. However, using LVLMs to generate descriptions often faces the challenge of object hallucination (OH), where the output text misrepresents actual objects in the input image. While previous studies attribute the occurrence of OH to the inclusion of more details, our study finds technical flaws in existing metrics, leading to unreliable evaluations of models and conclusions about OH. This has sparked a debate on the question: Do more details always introduce more hallucinations in LVLM-based image captioning? In this paper, we address this debate by proposing a novel decoding strategy, Differentiated Beam Decoding (DBD), along with a reliable new set of evaluation metrics: CLIP-Precision, CLIP-Recall, and CLIP-F1. DBD decodes the wealth of information hidden in visual input into distinct language representations called unit facts in parallel. This decoding is achieved via a well-designed differential score that guides the parallel search and candidate screening. The selected unit facts are then aggregated to generate the final caption. Our proposed metrics evaluate the comprehensiveness and accuracy of image captions by comparing the embedding groups of ground-truth image regions and generated text partitions. Extensive experiments on the Visual Genome dataset validate the effectiveness of our approach, demonstrating that it produces detailed descriptions while maintaining low hallucination levels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
