NeuroABench: 一种多模态评估基准，用于神经外科解剖识别
NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identification
AI总结：
NeuroABench是首个用于评估神经外科领域解剖理解的多模态基准，通过实验揭示了MLLMs在解剖识别任务中的局限性，并展示了人类表现与模型之间的差距。
摘要：
多模态大语言模型（MLLMs）在手术视频理解中展现出显著潜力。随着零样本性能的提升和更有效的机人类交互，它们为推进手术教育和辅助提供了坚实基础。然而，现有研究和数据集主要关注手术流程和工作的理解，而对解剖理解的关键作用关注有限。在临床实践中，外科医生依赖精确的解剖理解来解读、审查和学习手术视频。为填补这一空白，我们引入了神经外科解剖基准（NeuroABench），这是首个专门用于评估神经外科领域解剖理解的多模态基准。NeuroABench包含9小时的标注神经外科视频，涵盖89种不同的手术过程，并使用一种新的多模态标注流程和多个审查周期开发。该基准评估68种临床解剖结构的识别，提供了一个严格且标准化的框架来评估模型性能。在超过10种最先进的MLLMs上的实验揭示了显著的局限性，最佳表现的模型在解剖识别任务中的准确率仅为40.87%。为进一步评估该基准，我们提取数据集的子集并进行具有信息性的测试，与四位神经外科培训生进行测试。结果显示，最佳表现的学员准确率为56%，最低分为28%，平均分为46.5%。虽然最佳的MLLM表现与最低分的学员相当，但仍显著落后于该组的平均表现。这一比较突显了MLLMs在解剖理解方面的进步，以及在实现人类水平表现方面仍存在的巨大差距。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown significant potential in surgical video understanding. With improved zero-shot performance and more effective human-machine interaction, they provide a strong foundation for advancing surgical education and assistance. However, existing research and datasets primarily focus on understanding surgical procedures and workflows, while paying limited attention to the critical role of anatomical comprehension. In clinical practice, surgeons rely heavily on precise anatomical understanding to interpret, review, and learn from surgical videos. To fill this gap, we introduce the Neurosurgical Anatomy Benchmark (NeuroABench), the first multimodal benchmark explicitly created to evaluate anatomical understanding in the neurosurgical domain. NeuroABench consists of 9 hours of annotated neurosurgical videos covering 89 distinct procedures and is developed using a novel multimodal annotation pipeline with multiple review cycles. The benchmark evaluates the identification of 68 clinical anatomical structures, providing a rigorous and standardized framework for assessing model performance. Experiments on over 10 state-of-the-art MLLMs reveal significant limitations, with the best-performing model achieving only 40.87% accuracy in anatomical identification tasks. To further evaluate the benchmark, we extract a subset of the dataset and conduct an informative test with four neurosurgical trainees. The results show that the best-performing student achieves 56% accuracy, with the lowest scores of 28% and an average score of 46.5%. While the best MLLM performs comparably to the lowest-scoring student, it still lags significantly behind the group's average performance. This comparison underscores both the progress of MLLMs in anatomical understanding and the substantial gap that remains in achieving human-level performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
