VidText：面向视频文本理解的全面评估
VidText: Towards Comprehensive Evaluation for Video Text Understanding
AI总结：
提出了VidText基准，通过分层评估框架和配对感知推理任务全面评估视频文本理解能力，实验表明当前18个大型多模态模型在该任务上仍有巨大改进空间。
摘要：
视频中嵌入的视觉文本携带丰富的语义信息，这对于整体视频理解和局部人类动作的细粒度推理至关重要。然而，现有的视频理解基准在很大程度上忽略了文本信息，而专门的OCR基准仅限于静态图像，限制了它们捕捉文本与动态视觉上下文之间交互的能力。为了填补这一空白，我们提出了VidText，一个旨在对视频文本理解进行全面深入评估的新基准。VidText具有以下主要特征：1）它涵盖了广泛的现实场景并支持多语言内容，包含了视频文本自然出现的多种设置。2）它引入了一个包含视频级、片段级和实例级任务的分层评估框架，能够评估全局摘要和局部检索能力。3）该基准还引入了一组配对的感知推理任务，涵盖从视觉文本感知到文本与视觉信息之间的跨模态推理。对18个最先进的大型多模态模型（LMM）进行的广泛实验表明，当前模型在大多数任务上表现不佳，仍有巨大的改进空间。进一步的分析强调了模型内在因素（如输入分辨率和OCR能力）以及外部因素（包括辅助信息的使用和思维链推理策略）的影响。我们希望VidText能填补当前视频理解基准的空白，并为未来在动态环境中进行视频文本多模态推理研究奠定基础。
英文摘要：
Visual texts embedded in videos carry rich semantic information, which is crucial for both holistic video understanding and fine-grained reasoning about local human actions. However, existing video understanding benchmarks largely overlook textual information, while OCR-specific benchmarks are constrained to static images, limiting their ability to capture the interaction between text and dynamic visual contexts. To address this gap, we propose VidText, a new benchmark designed for comprehensive and in-depth evaluation of video text understanding. VidText offers the following key features: 1) It covers a wide range of real-world scenarios and supports multilingual content, encompassing diverse settings where video text naturally appears. 2) It introduces a hierarchical evaluation framework with video-level, clip-level, and instance-level tasks, enabling assessment of both global summarization and local retrieval capabilities. 3) The benchmark also introduces a set of paired perception reasoning tasks, ranging from visual text perception to cross-modal reasoning between textual and visual information. Extensive experiments on 18 state-of-the-art Large Multimodal Models (LMMs) reveal that current models struggle across most tasks, with significant room for improvement. Further analysis highlights the impact of both model-intrinsic factors, such as input resolution and OCR capability, and external factors, including the use of auxiliary information and Chain-of-Thought reasoning strategies. We hope VidText will fill the current gap in video understanding benchmarks and serve as a foundation for future research on multimodal reasoning with video text in dynamic environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
