CAPability: 一个全面的视觉描述基准，用于评估正确性和全面性
CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
AI总结：
CAPability是一个全面的视觉描述基准，通过12个维度评估描述的正确性和全面性，利用人类标注数据和启发式指标提升多模态模型的描述能力分析。
摘要：
视觉描述基准在现代多模态大语言模型（MLLMs）出现后变得过时了，因为简短的真实句子和传统指标无法有效评估详细的描述。尽管最近的基准试图通过关注关键词提取或以对象为中心的评估来解决这个问题，但它们仍然局限于模糊视图或对象视图分析和不完整的视觉元素覆盖。在本文中，我们介绍了CAPability，一个全面的多视角基准，用于在12个维度上评估视觉描述，涵盖六个关键视角。我们整理了近11000张由人类标注的图像和视频，并带有视觉元素注释，以评估生成的描述。CAPability通过精度和命中率指标稳定地评估描述的正确性和全面性。通过将注释转换为问答对，我们进一步引入了一个启发式指标，即“知道但无法说出”（K̃T），表明问答能力和描述能力之间存在显著的性能差距。我们的工作为MLLMs的描述能力提供了全面分析，因为我们识别了它们在不同维度上的优势和劣势，指导未来研究以增强其特定方面的能力。
英文摘要：
Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with \textit{precision} and \textit{hit} metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, \textit{know but cannot tell} ($K\bar{T}$), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
