arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.08071cs.CL
收藏

SurgiQ: 用于评估大语言模型手术理解的大规模多领域基准

SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi, Hamdan Alhadhrami, Preslav Nakov, Cesare Stefanini

AI总结:

提出SurgiQ基准,包含13,055道多选题,覆盖六个外科领域和四种题型,用于评估LLM的手术推理能力。实验显示最佳模型准确率仅68.1%,通用模型优于多数生物医学模型,表明当前医学专业化未能充分覆盖手术知识。

摘要:

大语言模型在外科领域的可靠评估仍不成熟。广泛的医学基准测试临床知识,而手术需要程序性推理、管理权衡、否定处理以及在合理手术决策中的选择。我们提出SurgiQ,一个纯文本、基于来源的基准,包含13,055道四选一多选题,涵盖六个外科领域和四种题型:基于案例、推理、最佳选项和否定题。SurgiQ通过多阶段生成、验证和专家审核流程,从外科教科书、开放获取论文和考试材料构建。我们在统一的log-likelihood协议下评估了35个开源权重LLM。结果显示仍有很大提升空间:较小模型通常接近25%的随机基线,而最佳模型达到68.1%的准确率。通用模型,尤其是Qwen2.5,优于大多数生物医学模型,表明当前的医学专业化尚未提供足够广泛的外科覆盖。校准和错误分析进一步表明,即使是强模型也会在临床合理的干扰项上犯自信的错误,这促使进行更可靠和更广泛的外科LLM评估。

英文摘要:

Reliable evaluation of large language models in surgery remains underdeveloped. Broad medical benchmarks test clinical knowledge, while surgery requires procedural reasoning, management trade-offs, negation handling, and selection among plausible operative decisions. We present SurgiQ, a text-only, source-grounded benchmark of 13,055 four-option multiple-choice questions spanning six surgical domains and four question formats: case-based, reasoning, best-option, and negative. SurgiQ is constructed from surgical textbooks, open-access papers, and examination material using a multi-stage generation, verification, and expert-audit pipeline. We evaluate 35 open-weight LLMs under a unified log-likelihood protocol. Our results show substantial remaining headroom: smaller models often remain near the 25\% random baseline, while the best model reaches 68.1\% accuracy. General-purpose models, especially Qwen2.5, outperform most biomedical models, suggesting that current medical specialization does not yet provide sufficiently broad surgical coverage. Calibration and error analysis further show that even strong models make confident mistakes on clinically plausible distractors, motivating more reliable and broader surgical LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑