While Large Language Models (LLMs) achieve expert-level performance on standard medical benchmarks through single-hop factual recall, they severely struggle with the complex, multi-hop diagnostic reasoning required in real-world clinical settings. A primary obstacle is "shortcut learning", where models exploit highly connected, generic hub nodes (e.g., "inflammation") in knowledge graphs to bypass authentic micro-pathological cascades. To address this, we introduce ShatterMed-QA, a bilingual benchmark of 10,558 multi-hop clinical questions designed to rigorously evaluate deep diagnostic reasoning. Our framework constructs a topology-regularized medical Knowledge Graph using a novel $k$-Shattering algorithm, which physically prunes generic hubs to explicitly sever logical shortcuts. We synthesize the evaluation vignettes by applying implicit bridge entity masking and topology-driven hard negative sampling, forcing models to navigate biologically plausible distractors without relying on superficial elimination. Comprehensive evaluations of 21 LLMs reveal massive performance degradation on our multi-hop tasks, particularly among domain-specific models. Crucially, restoring the masked evidence via Retrieval-Augmented Generation (RAG) triggers near-universal performance recovery, validating ShatterMed-QA's structural fidelity and proving its efficacy in diagnosing the fundamental reasoning deficits of current medical AI. Explore the dataset, interactive examples, and full leaderboards at our project website: https://shattermed-qa-web.vercel.app/
Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);School of Cyberspace Security, Southeast University(东南大学网络空间安全学院)
MuSiQue: Multihop Questions via Single-hop Question Composition
Stony Brook University(石溪大学);Allen Institute for AI(艾伦人工智能研究所)
多跳推理仍然是一个难以捉摸的目标,因为现有的多跳基准被认为是在很大程度上可以通过捷径解决。我们能否创建一个问答(QA)数据集,通过构造,\emph{requires}适当的多跳推理?为此,我们引入了一种自下而上的方法,该方法系统地选择连接的单跳问题的可组合对,即,其中一个推理步骤严重依赖于来自另一个推理步骤的信息。这种自下而上的方法使我们能够探索广阔的问题空间,并添加严格的过滤器以及其他针对关联推理的机制。它提供了对构造过程和生成的$k$跳问题的属性的细粒度控制。我们使用这种方法来创建MuSiQue-Ans,这是一个新的多跳QA数据集,包含25 K 2-4跳问题。相对于现有的数据集,MuSiQue-Ans总体上更困难(人机差距增加了3倍),并且更难通过断开推理进行欺骗(例如,单跳模型在F1中具有30点下降)。我们进一步添加无法回答的对比问题,以产生更严格的数据集MuSiQue-Full。我们希望我们的数据集能够帮助NLP社区开发执行真正多跳推理的模型。
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
Department of Artificial Intelligence, Chung-Ang University(Chung-Ang大学人工智能系);Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University(Chung-Ang大学高级成像科学研究院,多媒体与电影)
National Institute of Informatics(情报学研究所);Nantes Université(南特大学);The Graduate University for Advanced Studies(高级研究大学院大学);The University of Tokyo(东京大学);EPFL(洛桑联邦理工学院)
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Baidu Inc.(百度公司);University of California(加州大学);University of Science and Technology of China(中国科学技术大学)
MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Tsinghua University(清华大学)