以文字作画：通过基准与对齐学习提升细粒度图像描述
Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
AI总结：
本文提出细粒度图像描述基准DeCapBench、评估指标DCScore及自动反馈收集方法FeedQuill，通过偏好优化显著减少VLM幻觉并提升描述性能，超越GPT-4o。
摘要：
图像描述长期以来是视觉理解中的关键任务，视觉语言模型（VLMs）的最新进展显著增强了生成细粒度图像描述的能力。然而，由于评估指标过时和标注粗糙，细粒度图像描述的评估仍未被充分探索。本文提出DeCapBench以及一种新指标DCScore，专门为细粒度描述任务设计。DCScore通过将回答分解为最小的自足单元——称为原始信息单元——并逐个评估，来评估幻觉和细粒度全面性。我们的评估表明，DCScore比其他基于规则或基于模型的指标更接近人类判断。同时，DeCapBench在描述性任务上与VLM竞技场结果表现出高度相关性，超越了现有的视觉语言模型基准。此外，我们提出了一种自动细粒度反馈收集方法FeedQuill，用于基于我们先进指标的偏好优化，在自动生成的偏好数据上展现出稳健的泛化能力。在多个VLM上的大量实验表明，我们的方法不仅显著减少幻觉，还提升了在各种基准上的性能，在超越GPT-4o的同时实现了更优的细粒度描述性能。
英文摘要：
Image captioning has long been a pivotal task in visual understanding, with recent advancements in vision-language models (VLMs) significantly enhancing the ability to generate detailed image captions. However, the evaluation of detailed image captioning remains underexplored due to outdated evaluation metrics and coarse annotations. In this paper, we introduce DeCapBench along with a novel metric, DCScore, specifically designed for detailed captioning tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by deconstructing responses into the smallest self-sufficient units, termed primitive information units, and assessing them individually. Our evaluation shows that DCScore aligns more closely with human judgment than other rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high correlation with VLM arena results on descriptive tasks, surpassing existing benchmarks for vision-language models. Additionally, we present an automatic fine-grained feedback collection method, FeedQuill, for preference optimization based on our advanced metric, showing robust generalization capabilities across auto-generated preference data. Extensive experiments on multiple VLMs demonstrate that our method not only significantly reduces hallucinations but also enhances performance across various benchmarks, achieving superior detail captioning performance while surpassing GPT-4o.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
