基于蒙特卡洛树搜索的视频字幕任务多模态大语言模型评估
Evaluating Multimodal Large Language Models on Video Captioning via Monte Carlo Tree Search
AI总结：
针对现有视频字幕评估基准的缺陷，研究提出基于MCTS的AutoCaption框架构建细粒度基准MCTS-VCB，评估20余款MLLMs并验证了其生成数据的微调增益效果。
摘要：
视频字幕可用于评估多模态大语言模型（MLLMs）的视频理解能力。然而，现有基准和评估协议存在关键问题，例如关键点生成不足或同质化、数据制作成本过高，以及评估范围有限。为解决这些问题，我们提出一个名为AutoCaption的自动化框架，该框架利用蒙特卡洛树搜索（MCTS）以迭代方式构建大量多样化的描述句（即关键点），全面表征视频内容。这种迭代字幕策略能够持续丰富动作、物体属性、环境细节等视频细节。我们应用AutoCaption构建了MCTS-VCB——一个覆盖视频细节的细粒度视频字幕基准，从而实现对MLLMs视频字幕任务能力的全面评估。我们在MCTS-VCB上评估了20余款不同规模的开源和闭源MLLMs。结果表明，MCTS-VCB能够有效且全面地评估视频字幕能力，其中Gemini-1.5-Pro取得了71.2的最高F1分数。有趣的是，我们使用AutoCaption生成的数据微调InternVL2.5-8B，帮助该模型在MCTS-VCB上实现了25.0%的整体提升，在DREAM-1K上实现了16.3%的提升，进一步证明了AutoCaption的有效性。代码和数据可在https://github.com/tjunlp-lab/MCTS-VCB 获取。
英文摘要：
Video captioning can be used to assess the video understanding capabilities of Multimodal Large Language Models (MLLMs). However, existing benchmarks and evaluation protocols suffer from crucial issues, such as inadequate or homogeneous creation of key points, exorbitant cost of data creation, and limited evaluation scopes. To address these issues, we propose an automatic framework, named AutoCaption, which leverages Monte Carlo Tree Search (MCTS) to construct numerous and diverse descriptive sentences (\textit{i.e.}, key points) that thoroughly represent video content in an iterative way. This iterative captioning strategy enables the continuous enhancement of video details such as actions, objects' attributes, environment details, etc. We apply AutoCaption to curate MCTS-VCB, a fine-grained video caption benchmark covering video details, thereby enabling a comprehensive evaluation of MLLMs on the video captioning task. We evaluate more than 20 open- and closed-source MLLMs of varying sizes on MCTS-VCB. Results show that MCTS-VCB can effectively and comprehensively evaluate the video captioning capability, with Gemini-1.5-Pro achieving the highest F1 score of 71.2. Interestingly, we fine-tune InternVL2.5-8B with the AutoCaption-generated data, which helps the model achieve an overall improvement of 25.0% on MCTS-VCB and 16.3% on DREAM-1K, further demonstrating the effectiveness of AutoCaption. The code and data are available at https://github.com/tjunlp-lab/MCTS-VCB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
