ComiCap：用于漫画画格密集描述的VLMs流水线
ComiCap: A VLMs pipeline for dense captioning of Comic Panels
AI总结：
提出一种利用视觉语言模型的漫画画格密集描述流水线，通过属性保留指标和密集标注测试集评估选择最佳模型，无需额外训练即可生成优于专门训练模型的带边界框描述，并已标注超200万个画格。
摘要：
随着单页和多页分析及合成模型的发展，漫画领域正在迅速进步。近期引入了基准和数据集来支持和评估模型在检测（画格、角色、文本）、链接（角色重识别和说话者识别）以及漫画元素分析（例如对话转录）等任务中的能力。然而，为了提供对故事情节的全面理解，模型不仅需要提取元素，还必须理解它们的关系并生成信息量极高的描述。在这项工作中，我们提出了一种利用视觉语言模型（VLMs）来获取密集且有依据的描述的流水线。为了构建我们的流水线，我们引入了一种属性保留指标，用于评估描述中是否识别了所有重要属性。此外，我们创建了一个密集标注的测试集，以公平地评估开源VLMs，并根据我们的指标选择最佳描述模型。我们的流水线生成的带有边界框的密集描述在定量和定性上均优于专门训练的模型所生成的描述，且无需任何额外训练。使用该流水线，我们标注了13,000本书中超过200万个画格，这些数据将在项目页面 https://github.com/emanuelevivoli/ComiCap 上提供。
英文摘要：
The comic domain is rapidly advancing with the development of single- and multi-page analysis and synthesis models. Recent benchmarks and datasets have been introduced to support and assess models' capabilities in tasks such as detection (panels, characters, text), linking (character re-identification and speaker identification), and analysis of comic elements (e.g., dialog transcription). However, to provide a comprehensive understanding of the storyline, a model must not only extract elements but also understand their relationships and generate highly informative captions. In this work, we propose a pipeline that leverages Vision-Language Models (VLMs) to obtain dense, grounded captions. To construct our pipeline, we introduce an attribute-retaining metric that assesses whether all important attributes are identified in the caption. Additionally, we created a densely annotated test set to fairly evaluate open-source VLMs and select the best captioning model according to our metric. Our pipeline generates dense captions with bounding boxes that are quantitatively and qualitatively superior to those produced by specifically trained models, without requiring any additional training. Using this pipeline, we annotated over 2 million panels across 13,000 books, which will be available on the project page https://github.com/emanuelevivoli/ComiCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
