全景描述：图像与文本之间的等价桥梁
Panoptic Captioning: An Equivalence Bridge for Image and Text
AI总结：
本文提出全景描述任务，通过PancapEngine和PancapChain方法提升图像与文本的等价描述能力，实验表明其优于现有大语言模型。
摘要：
本文引入了全景描述，一种新的任务，旨在寻找图像的最小文本等价物，具有广泛的应用潜力。我们通过将该任务定义为生成图像的综合文本描述，迈出了解决全景描述的第一步。该描述涵盖了所有实体、各自的位置和属性、实体之间的关系以及全局图像状态。通过广泛的评估，我们的工作揭示了最先进的多模态大语言模型（MLLMs）在解决全景描述任务上表现有限。为此，我们提出了一种有效的数据引擎，名为PancapEngine，用于生成高质量数据，并提出了一种新的方法，名为PancapChain，以提高全景描述能力。具体而言，我们的PancapEngine首先通过精心设计的检测套件检测图像中的各种实体类别，然后使用实体感知提示生成所需的全景描述。此外，我们的PancapChain明确将具有挑战性的全景描述任务分解为多个阶段，并逐步生成全景描述。更重要的是，我们贡献了一个全面的指标，名为PancapScore，以及一个由人类整理的测试集，用于可靠的模型评估。实验表明，我们的PancapChain-13B模型能够击败最先进的开源MLLMs，如InternVL-2.5-78B，并且甚至超越了专有模型，如GPT-4o和Gemini-2.0-Pro，证明了我们数据引擎和方法的有效性。项目页面：https://visual-ai.github.io/pancap/
英文摘要：
This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
