CRiT-QA：利用反事实链和干扰陷阱评估多跳推理
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
浏览论文内容
中文总结 AI 辅助
研究大语言模型多跳推理能力评估难题，提出CRiT-QA数据集，通过反事实实体转换推理链、注入干扰链解决模型依赖知识和利用捷径问题，实验表明该数据集能暴露模型弱点，为评估多跳推理及开发可靠模型提供基础。
中文摘要 AI 辅助
评估大语言模型的多跳推理能力仍然是一项重大挑战。尽管当前模型在现有多跳问答数据集上取得了不错的结果，但这种性能往往掩盖了两个关键漏洞：依赖内部参数知识而非遵循给定上下文，以及利用数据集捷径。我们引入了CRiT-QA数据集来解决这两个限制。它通过反事实实体转换事实推理链以消除对记忆知识的依赖并加强严格的上下文依赖，还注入多锚干扰链。实验表明，与标准数据集相比，大语言模型在CRiT-QA上性能大幅下降，这暴露了它们在反事实条件和干扰陷阱方面的脆弱性。CRiT-QA为评估真正的多跳推理提供了严格的诊断工具，并为开发更可靠、基于证据的大语言模型奠定了基础。
英文摘要
Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge. Although current models achieve strong results on existing multi-hop question answering datasets, such performance often masks two critical vulnerabilities: (1) reliance on internal parametric knowledge rather than adherence to the provided context, and (2) exploitation of dataset shortcuts, such as single-document cues or type-matching, that diminish the need for genuine evidence aggregation across multiple documents. We introduce CRiT-QA (Counterfactual Reasoning with Traps), a dataset explicitly designed to address both limitations. To neutralize reliance on memorized knowledge and enforce strict context dependency, CRiT-QA transforms factual reasoning chains with counterfactual entities. Furthermore, it injects multi-anchor distractor chains, plausible but incorrect reasoning paths that diverge at different hops. These traps require models to follow the entire reasoning process rather than exploiting shallow heuristics. Our experiments show that LLMs exhibit substantial performance degradation on CRiT-QA compared to standard datasets, exposing their vulnerability to counterfactual conditions and distractor traps. CRiT-QA thus serves as a rigorous diagnostic tool for evaluating genuine multi-hop reasoning and provides a foundation for developing more reliable, evidence-grounded LLMs.
发表机构
- Department of Artificial Intelligence, Chung-Ang University（Chung-Ang大学人工智能系）
- Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University（Chung-Ang大学高级成像科学研究院，多媒体与电影）
机构由 AI 辅助整理，请以论文原文为准。
