CaReBench：面向视频描述与检索的细粒度基准
CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval
AI总结：
本文提出含1,000对视频与详细人工字幕的 CaReBench 基准及 ReBias、CapST 指标，并以两阶段 SFT 训练 MLLM 基线，实现检索与描述的统一评估。
摘要：
视频理解，包括视频描述和检索，对视频-语言模型（VLM）而言仍是一项巨大挑战。现有的视频检索和描述基准只包含简短描述，限制了其评估详细视频理解的能力。为解决这一问题，我们提出 CaReBench，这是一个用于细粒度视频描述和检索的测试基准，包含 1,000 对高质量视频与人工标注的详细字幕。其独特之处在于，它为每个视频提供人工分离的空间标注和时间标注。基于这一设计，我们引入 ReBias 和 CapST 两个评估指标，分别专门针对视频检索和视频描述任务。这些指标能够全面考察 VLM 内在的空间与时间偏差。此外，为在统一框架中同时处理视频检索和视频描述任务，我们开发了一个基于多模态语言模型（MLLM）的简单基线。通过实施两阶段监督微调（SFT），我们充分释放 MLLM 的潜力，使其不仅能够生成详细的视频描述，还能提取视频特征。令人惊讶的是，实验结果表明，与为检索设计的基于 CLIP 的模型以及擅长视频描述的流行 MLLM 相比，我们的基线在细粒度视频检索和视频详细描述两方面均表现出具有竞争力的性能。
英文摘要：
Video understanding, including video captioning and retrieval, is still a great challenge for video-language models (VLMs). The existing video retrieval and caption benchmarks only include short descriptions, limits their ability of detailed video understanding evaluation. To address this problem, we present CaReBench, a testing benchmark for fine-grained video captioning and retrieval with 1,000 high-quality pairs of videos and human-annotated detailed captions. Uniquely, it provides manually separated spatial annotations and temporal annotations for each video. Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks, respectively. These metrics enable a comprehensive investigation into the spatial and temporal biases inherent in VLMs. In addition, to handle both video retrieval and video captioning tasks in a unified framework, we develop a simple baseline based on a Multimodal Language Model (MLLM). By implementing a two-stage Supervised Fine-Tuning (SFT), we fully unlock the potential of MLLM, enabling it not only to generate detailed video descriptions but also to extract video features. Surprisingly, experimental results demonstrate that, compared to the CLIP-based models designed for retrieval and the popular MLLMs skilled in video captioning, our baseline shows competitive performance in both fine-grained video retrieval and video detailed captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
