SciVideoBench：大型多模态模型科学视频推理基准
SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models
AI总结：
针对现有视频基准推理简单、性能饱和的问题，提出SciVideoBench基准，含1,000道科学实验视频选择题，评估LMMs高级推理，发现Gemini 2.5 Pro等模型性能缺陷，为未来发展指明方向。
摘要：
大型多模态模型（LMMs）在各种能力上取得了显著进展；然而，科学领域中复杂的视频推理仍然是一个重大且具有挑战性的前沿问题。当前的视频基准主要针对普遍依赖感知/识别的通用场景，而推理任务相对简单，导致性能饱和，从而无法有效评估高级多模态认知技能。为填补这一关键空白，我们引入了SciVideoBench，这是一个专门设计用于评估科学背景下高级视频推理能力的严格基准。SciVideoBench包含1,000道精心设计的选择题，这些题目源自覆盖超过25个专业学术学科的前沿科学实验视频，并经过半自动系统验证。每道题都需要复杂的领域特定知识、精确的时空感知以及细致的逻辑推理，有效挑战模型的高阶认知能力。我们的评估揭示了最先进的专有和开源LMMs（包括Gemini 2.5 Pro和Qwen2.5-VL）存在显著的性能缺陷，表明视频推理能力仍有很大的提升空间。对推理复杂性和视觉基础等关键因素的详细分析为LMMs的未来发展提供了宝贵的见解和明确方向，推动真正有能力的多模态AI共同科学家的演进。我们希望SciVideoBench能够符合社区的兴趣，并帮助推动前沿AI在更广泛科学领域的边界。
英文摘要：
Large Multimodal Models (LMMs) have achieved remarkable progress across various capabilities; however, complex video reasoning in the scientific domain remains a significant and challenging frontier. Current video benchmarks predominantly target general scenarios where perception/recognition is heavily relied on, while with relatively simple reasoning tasks, leading to saturation and thus failing to effectively evaluate advanced multimodal cognitive skills. To address this critical gap, we introduce SciVideoBench, a rigorous benchmark specifically designed to assess advanced video reasoning in scientific contexts. SciVideoBench consists of 1,000 carefully crafted multiple-choice questions derived from cutting-edge scientific experimental videos spanning over 25 specialized academic subjects and verified by a semi-automatic system. Each question demands sophisticated domain-specific knowledge, precise spatiotemporal perception, and intricate logical reasoning, effectively challenging models' higher-order cognitive abilities. Our evaluation highlights significant performance deficits in state-of-the-art proprietary and open-source LMMs, including Gemini 2.5 Pro and Qwen2.5-VL, indicating substantial room for advancement in video reasoning capabilities. Detailed analyses of critical factors such as reasoning complexity and visual grounding provide valuable insights and clear direction for future developments in LMMs, driving the evolution of truly capable multimodal AI co-scientists. We hope SciVideoBench could fit the interests of the community and help to push the boundary of cutting-edge AI for border science.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
