ScaleCap：基于双模态去偏的推理时可扩展图像描述生成
ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing
AI总结：
本文提出ScaleCap，一种推理时可扩展的图像描述策略，通过启发式问答和对比句子评分两个组件，分别解决LVLMs的多模态偏差和语言偏差问题，生成更准确、均衡且信息丰富的描述，在11个基准上实现持续性能提升。
摘要：
本文提出了ScaleCap，一种推理时可扩展的图像描述生成策略，能够生成全面且详细的图像描述。高质量图像描述生成的关键挑战在于LVLMs固有的偏差：多模态偏差导致描述粒度不均衡，对某些元素提供详细描述而对其他元素仅一笔带过；语言偏差导致对不存在对象的幻觉描述。为解决这些问题，我们提出了一种可扩展的去偏描述策略，随着推理预算的增加，持续丰富和校准描述。具体而言，我们提出了两个新颖的组件：启发式问答和对比句子评分。前者基于图像生成内容特定的问题并回答，逐步将相关信息注入描述中。后者采用句子级离线对比解码，有效识别和消除由语言偏差引起的幻觉。随着推理成本增加，ScaleCap提出更多启发式问题，逐步捕捉额外视觉细节，生成更准确、均衡且信息丰富的描述。广泛的模态对齐实验证明了ScaleCap的有效性。使用ScaleCap标注450K图像并将其用于LVLM预训练，在11个广泛使用的基准上实现了持续的性能提升。此外，ScaleCap在两项额外任务中展示了生成描述的卓越丰富度和保真度：在VQA任务中用描述替换图像，以及从描述重建图像以评估语义覆盖。代码可在https://github.com/Cooperx521/ScaleCap获取。
英文摘要：
This paper presents ScaleCap, an inference-time scalable image captioning strategy that generates comprehensive and detailed image captions. The key challenges of high-quality image captioning lie in the inherent biases of LVLMs: multimodal bias resulting in imbalanced descriptive granularity, offering detailed accounts of some elements while merely skimming over others; linguistic bias leading to hallucinated descriptions of non-existent objects. To address these issues, we propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget. Specifically, we propose two novel components: heuristic question answering and contrastive sentence rating. The former generates content-specific questions based on the image and answers them to progressively inject relevant information into the caption. The latter employs sentence-level offline contrastive decoding to effectively identify and eliminate hallucinations caused by linguistic biases. With increased inference cost, more heuristic questions are raised by ScaleCap to progressively capture additional visual details, generating captions that are more accurate, balanced, and informative. Extensive modality alignment experiments demonstrate the effectiveness of ScaleCap. Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used benchmarks. Furthermore, ScaleCap showcases superb richness and fidelity of generated captions with two additional tasks: replacing images with captions in VQA task, and reconstructing images from captions to assess semantic coverage. Code is available at https://github.com/Cooperx521/ScaleCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
