SiMing-Bench：从连续交互评估临床技能视频中的过程正确性
SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
AI总结：
SiMing-Bench旨在评估多模态大语言模型在临床技能视频中通过连续交互更新过程状态的能力，通过医师标注的数据集和标准化评分标准，揭示模型在过程级判断上的不足。
摘要：
当前多模态大语言模型（MLLM）视频基准主要关注事件识别、时间顺序和长上下文回忆，但忽略了专家过程判断所需的能力：跟踪持续交互如何更新过程状态并决定后续动作的正确性。我们引入SiMing-Bench，首个针对完整临床技能视频评估该能力的基准。它旨在评估交互驱动的状态更新是否在整个工作流程中保持过程正确性。SiMing-Bench通过SiMing-Score实现，包含涵盖心肺复苏、自动体外除颤器操作和袋面罩通气的实临床技能考试视频数据集，每种视频均配以标准化的分步评分标准和双专家标签。在多样化的开源和闭源MLLM上，我们观察到与医师判断一致性的持续薄弱。此外，即使整体流程级相关性看起来可接受，对评分标准定义的中间步骤的弱表现仍持续存在，表明粗粒度全局评估严重高估了当前模型的过程判断能力。进一步分析二元步骤判断和步骤对齐片段表明，瓶颈不在于细粒度评分或时间定位，而在于建模连续交互如何随时间更新过程状态。
英文摘要：
Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but overlook a harder capability required for expert procedural judgment: tracking how ongoing interactions update the procedural state and thereby determine the correctness of later actions. We introduce SiMing-Bench, the first benchmark for evaluating this capability from full-length clinical skill videos. It targets rubric-grounded process-level judgment of whether interaction-driven state updates preserve procedural correctness across an entire workflow. SiMing-Bench is instantiated with SiMing-Score, a physician-annotated dataset of real clinical skill examination videos spanning cardiopulmonary resuscitation, automated external defibrillator operation, and bag-mask ventilation, each paired with a standardized step-wise rubric and dual-expert labels. Across diverse open- and closed-source MLLMs, we observe consistently weak agreement with physician judgments. Moreover, weak performance on rubric-defined intermediate steps persists even when overall procedure-level correlation appears acceptable, suggesting that coarse global assessment substantially overestimates current models' procedural judgment ability. Additional analyses with binary step judgment and step-aligned clips indicate that the bottleneck is not merely fine-grained scoring or temporal localization, but modeling how continuous interactions update procedural state over time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
