arXiv 2603.199932026-03-20cs.CV

MedSPOT: 一种面向临床GUI的流程感知序列接地基准

MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

Gaash Research Lab, National Institute of Technology Srinagar, India（加什研究实验室，印度锡里纳杰尔国家理工学院）；e\& Group, UAE（e&集团，阿联酋）；Mohammad Bin Zayed University of Artificial Intelligence (MBZUAI), UAE（穆罕默德·本·扎耶德人工智能大学（MBZUAI），阿联酋）；King Abdullah University of Science and Technology (KAUST), Saudi Arabia（国王 Abdullah 科学技术大学（KAUST），沙特阿拉伯）

Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf

备注：: Project page: https://rozainmalik.github.io/MedSPOT_web/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

MedSPOT针对临床GUI中需流程驱动的序列推理问题，通过216个任务驱动视频和597个标注关键帧，评估多模态模型在医疗软件中的可靠性与安全性。

摘要：

尽管多模态大语言模型（MLLMs）在视觉接地方面取得显著进展，但其在高风险临床软件环境中的可靠性仍待探索。现有GUI基准主要关注孤立的单步接地查询，忽视了真实医疗界面中任务在独立步骤间演变的序列性与流程驱动性。我们引入MedSPOT，一种面向临床GUI的流程感知序列接地基准。与以往将接地视为独立预测任务的基准不同，MedSPOT将程序交互建模为结构化的空间决策序列。该基准包含216个任务驱动视频和597个标注关键帧，每个任务包含2至3个相互依赖的接地步骤。此设计捕捉了界面层次、上下文依赖性和在动态变化条件下精细的空间精度。为评估程序鲁棒性，我们提出了一种严格的序列评估协议，在首次错误接地预测时终止任务评估，明确测量多步骤工作流中的误差传播。我们进一步引入了全面的失败分类，包括边缘偏差、小目标错误、无预测、近似命中、远距离命中和工具栏混淆，以系统诊断临床GUI中的模型行为。通过将评估从孤立接地转向流程感知的序列推理，MedSPOT建立了评估多模态模型在医疗软件环境中的现实且安全关键的基准。代码和数据可在：https://github.com/Tajamul21/MedSPOT 获取。

英文摘要：

Despite the rapid progress of Multimodal Large Language Models (MLLMs), their ability to perform reliable visual grounding in high-stakes clinical software environments remains underexplored. Existing GUI benchmarks largely focus on isolated, single-step grounding queries, overlooking the sequential, workflow-driven reasoning required in real-world medical interfaces, where tasks evolve across independent steps and dynamic interface states. We introduce MedSPOT, a workflow-aware sequential grounding benchmark for clinical GUI environments. Unlike prior benchmarks that treat grounding as a standalone prediction task, MedSPOT models procedural interaction as a sequence of structured spatial decisions. The benchmark comprises 216 task-driven videos with 597 annotated keyframes, in which each task consists of 2 to 3 interdependent grounding steps within realistic medical workflows. This design captures interface hierarchies, contextual dependencies, and fine-grained spatial precision under evolving conditions. To evaluate procedural robustness, we propose a strict sequential evaluation protocol that terminates task assessment upon the first incorrect grounding prediction, explicitly measuring error propagation in multi-step workflows. We further introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to enable systematic diagnosis of model behavior in clinical GUI settings. By shifting evaluation from isolated grounding to workflow-aware sequential reasoning, MedSPOT establishes a realistic and safety-critical benchmark for assessing multimodal models in medical software environments. Code and data are available at: https://github.com/Tajamul21/MedSPOT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。