arXiv 2603.013432026-03-02cs.CLcs.AI

PanCanBench: 用于评估大型语言模型在胰腺肿瘤学中的综合基准

PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

Department of Biostatistics, University of Washington（华盛顿大学生物统计学系）；Clinical Research Division, Fred Hutch Cancer Center（Fred Hutch癌症中心临床研究部）；Division of Hematology and Oncology, Department of Medicine, University of Washington（华盛顿大学医学系血液学与肿瘤学分会）；Allen Institute for AI（Allen人工智能研究所）；Department of Computer Science and Engineering, University of Washington（华盛顿大学计算机科学与工程系）；Public Health Sciences, Biostatistics, Fred Hutchinson Cancer Center（Fred Hutchinson癌症中心公共卫生科学与生物统计学）

Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, B… 展开作者

Yimin Zhao, Sheela R. Damle, Simone E. Dekker, Scott Geng, Karly Williams Silva, Jesse J Hubbard, Manuel F Fernandez, Fatima Zelada-Arenas, Alejandra Alvarez, Brianne Flores, Alexis Rodriguez, Stephen Salerno, Carrie Wright, Zihao Wang, Pang Wei Koh, Jeffrey T. Leek

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PanCanBench通过评估22种LLM在胰腺肿瘤学问题上的表现，揭示了模型在事实准确性、临床完整性和网络搜索整合方面的差异。

摘要：

大型语言模型（LLMs）在标准化考试中已达到专家水平表现，但多项选择题的准确性并不能反映真实世界中的临床效用和安全性。随着患者和医生越来越多地使用LLMs对复杂状况如胰腺癌进行指导，评估必须超越一般的医学知识。现有的框架，如HealthBench，依赖于模拟查询且缺乏疾病特定的深度。此外，基于高评分标准的分数并不能保证事实的正确性，这凸显了评估幻觉的必要性。我们开发了一个人机协同的管道，以创建专家评分标准，用于从胰腺癌行动网络（PanCAN）获取的去标识患者问题。所得到的基准，PanCanBench，包括282个真实患者问题中的3130个问题特定标准。我们使用LLM-as-a-judge框架评估了22个专有和开源的LLMs，测量临床完整性、事实准确性以及网络搜索整合。模型在评分标准下的完整性表现出显著差异，得分范围从46.5%到82.3%。事实错误很常见，幻觉率（响应中包含至少一个事实错误的百分比）从Gemini-2.5 Pro和GPT-4o的6.0%到Llama-3.1-8B的53.8%。重要的是，新的推理优化模型并未一致地提高事实准确性：虽然o3获得了最高的评分标准分数，但其产生的不准确性比其他GPT家族模型更频繁。网络搜索整合并不能保证更好的响应。当启用网络搜索时，Gemini-2.5 Pro的平均得分从66.8%降至63.9%，GPT-5的平均得分从73.8%降至72.8%。合成的人工智能生成的评分标准平均提高了17.9个分数点，同时通常保持相似的相对排名。

英文摘要：

Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety. As patients and clinicians increasingly use LLMs for guidance on complex conditions such as pancreatic cancer, evaluation must extend beyond general medical knowledge. Existing frameworks, such as HealthBench, rely on simulated queries and lack disease-specific depth. Moreover, high rubric-based scores do not ensure factual correctness, underscoring the need to assess hallucinations. We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions from the Pancreatic Cancer Action Network (PanCAN). The resulting benchmark, PanCanBench, includes 3,130 question-specific criteria across 282 authentic patient questions. We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration. Models showed substantial variation in rubric-based completeness, with scores ranging from 46.5% to 82.3%. Factual errors were common, with hallucination rates (the percentages of responses containing at least one factual error) ranging from 6.0% for Gemini-2.5 Pro and GPT-4o to 53.8% for Llama-3.1-8B. Importantly, newer reasoning-optimized models did not consistently improve factuality: although o3 achieved the highest rubric score, it produced inaccuracies more frequently than other GPT-family models. Web-search integration did not inherently guarantee better responses. The average score changed from 66.8% to 63.9% for Gemini-2.5 Pro and from 73.8% to 72.8% for GPT-5 when web search was enabled. Synthetic AI-generated rubrics inflated absolute scores by 17.9 points on average while generally maintaining similar relative ranking.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。