FINECAPTION：任意粒度下聚焦你所指区域的组合式图像描述生成
FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity
AI总结：
针对VLMs在细粒度区域组合信息感知上的不足，本文提出FINECAPTION模型，可识别任意掩码并处理高分辨率图像，实现多粒度组合式图像描述，并构建了COMPOSITIONCAP数据集，实验证明其优于现有最先进方法。
摘要：
大型视觉语言模型（VLMs）的出现显著推动了多模态任务的发展，使得在图像和视频描述生成、视觉问答以及跨模态检索等各种应用中能够进行更复杂、更准确的推理。尽管VLMs能力卓越，但它们在细粒度图像区域组合信息的感知方面仍存在困难。具体而言，它们难以将分割掩码与相应语义准确对齐，也难以精确描述所指区域的组合特性。然而，组合性——即理解和生成已知视觉与文本成分的新组合的能力——对于VLMs促进跨模态的连贯推理和理解至关重要。为了解决这一问题，我们提出了FINECAPTION，这是一种新颖的VLM，能够将任意掩码识别为指称输入，并处理高分辨率图像，从而在不同粒度级别上进行组合式图像描述生成。为支持这一工作，我们引入了COMPOSITIONCAP，这是一个用于多粒度区域组合式图像描述生成的新数据集，该数据集提出了组合属性感知的区域图像描述生成任务。实证结果表明，与其他最先进的VLMs相比，我们提出的模型具有有效性。此外，我们分析了当前VLMs在识别各种视觉提示以进行组合式区域图像描述生成方面的能力，指出了VLM设计和训练中需要改进的方面。
英文摘要：
The advent of large Vision-Language Models (VLMs) has significantly advanced multimodal tasks, enabling more sophisticated and accurate reasoning across various applications, including image and video captioning, visual question answering, and cross-modal retrieval. Despite their superior capabilities, VLMs struggle with fine-grained image regional composition information perception. Specifically, they have difficulty accurately aligning the segmentation masks with the corresponding semantics and precisely describing the compositional aspects of the referred regions. However, compositionality - the ability to understand and generate novel combinations of known visual and textual components - is critical for facilitating coherent reasoning and understanding across modalities by VLMs. To address this issue, we propose FINECAPTION, a novel VLM that can recognize arbitrary masks as referential inputs and process high-resolution images for compositional image captioning at different granularity levels. To support this endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region compositional image captioning, which introduces the task of compositional attribute-aware regional image captioning. Empirical results demonstrate the effectiveness of our proposed model compared to other state-of-the-art VLMs. Additionally, we analyze the capabilities of current VLMs in recognizing various visual prompts for compositional region image captioning, highlighting areas for improvement in VLM design and training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
