arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.12458cs.CLcs.AI
收藏

打破捷径:一种用于LLMs多跳医学推理的拓扑正则化基准

Shattering the Shortcut: A Topology-Regularized Benchmark for Multi-hop Medical Reasoning in LLMs

University of Technology Sydney(悉尼技术大学)

Xing Zi, Xinying Zhou, Jinghao Xiao, Catarina Moreira, Mukesh Prasad

AI总结:

本文提出ShatterMed-QA基准,通过拓扑正则化知识图谱评估深度诊断推理能力,揭示LLMs在多跳医学推理中的缺陷,并验证RAG方法的有效性。

摘要:

尽管大型语言模型(LLMs)在标准医学基准上通过单跳事实回忆达到专家水平,但在真实临床环境中复杂的多跳诊断推理任务中表现不佳。主要障碍是

英文摘要:

While Large Language Models (LLMs) achieve expert-level performance on standard medical benchmarks through single-hop factual recall, they severely struggle with the complex, multi-hop diagnostic reasoning required in real-world clinical settings. A primary obstacle is "shortcut learning", where models exploit highly connected, generic hub nodes (e.g., "inflammation") in knowledge graphs to bypass authentic micro-pathological cascades. To address this, we introduce ShatterMed-QA, a bilingual benchmark of 10,558 multi-hop clinical questions designed to rigorously evaluate deep diagnostic reasoning. Our framework constructs a topology-regularized medical Knowledge Graph using a novel $k$-Shattering algorithm, which physically prunes generic hubs to explicitly sever logical shortcuts. We synthesize the evaluation vignettes by applying implicit bridge entity masking and topology-driven hard negative sampling, forcing models to navigate biologically plausible distractors without relying on superficial elimination. Comprehensive evaluations of 21 LLMs reveal massive performance degradation on our multi-hop tasks, particularly among domain-specific models. Crucially, restoring the masked evidence via Retrieval-Augmented Generation (RAG) triggers near-universal performance recovery, validating ShatterMed-QA's structural fidelity and proving its efficacy in diagnosing the fundamental reasoning deficits of current medical AI. Explore the dataset, interactive examples, and full leaderboards at our project website: https://shattermed-qa-web.vercel.app/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑