不遗漏任何细节：重新审视细粒度图像描述中的自检索方法
No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
AI总结：
针对细粒度图像描述中自检索微调导致描述失真与幻觉的问题，本文提出Visual Caption Boosting框架和BagCurri训练课程，在保持忠实度的同时提升细粒度描述能力，并构建TrueMatch基准评估细微视觉差异捕捉能力，以更少参数超越现有开源MLLM。
摘要：
图像描述系统无法生成细粒度描述，因为其训练数据要么存在噪声（替代文本），要么过于泛化（人工标注）。最大似然训练进一步加剧了这一问题，因为它鼓励生成频繁出现的短语。此前的工作尝试通过使用自检索（SR）奖励对描述器进行微调来解决这一局限。然而，我们发现SR微调倾向于降低描述的忠实度，甚至产生幻觉。在本工作中，我们通过改进描述系统的MLE初始化并为SR微调过程设计课程来绕过这一瓶颈。为此，我们提出：（1）Visual Caption Boosting，一个新颖的框架，在保持锚定于人工标注的同时，向通用图像描述数据集注入细粒度性；（2）BagCurri，一个精心设计的训练课程，能更优化地利用自检索奖励的对比性质。两者结合，使描述器能够描述图像中的细粒度方面，同时保持对真实描述（ground-truth captions）的忠实度。我们的方法在针对99个随机干扰项（RD100）的自检索上比此前工作高出+8.9%（Dessi et al., 2023）；在ImageCoDe上高出+7.6%。此外，现有的评估描述系统的指标未能奖励多样性，也无法评估模型的细粒度理解能力。我们的第三项贡献通过从评估的视角提出自检索来解决这一问题。我们引入了TrueMatch，一个由高度相似图像包组成的基准，利用SR来评估描述器捕捉细微视觉差异的能力。我们在TrueMatch上评估并比较了多个最先进的开源MLLM，发现我们的SR方法以显著优势超越它们所有（例如，比Cambrian高+4.8% - 7.1%），同时参数量少1-2个数量级。
英文摘要：
Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
