arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20128cs.CL
收藏

MixRea: 在大型语言模型中评估显式-隐式推理的基准测试

MixRea: Benchmarking Explicit-Implicit Reasoning in Large Language Models

DeepMind;DeepSeek-AI;Anthropic;Qwen;Touvron;Team

Yuanqing Cai, Ziyi Huang, Minhao Liu, Lixin Duan, Wen Li, Yanru Zhang

AI总结:

本文提出MixRea基准测试,用于评估大型语言模型在显式和隐式推理任务中的表现,发现即使最佳模型也存在注意力不足的问题,并提出PRCP方法来改进推理能力。

摘要:

大型语言模型(LLMs)正越来越多地融入高风险决策中。受人类认知中'注意力盲区'理论的启发,我们研究LLMs是否在显式任务指令下表现出类似限制:未能关注到细微但重要的上下文线索。为此,我们引入显式-隐式推理任务,并提出MixRea基准测试,包含2246道多选题,覆盖9种推理类型,显式和隐式信息分布各异。对21种先进LLMs的评估显示,即使最佳推理模型(Gemini 2.5 Pro)也只能达到42.8%的一致性,揭示了普遍存在的注意力盲区。为缓解这一问题,我们提出潜在关系完成提示(PRCP),一种通过恢复被忽视的因果关系来提升推理能力的提示方法。进一步分析显示,这一限制在多样化的多源推理任务中依然存在,凸显了对更认知对齐模型的需求。

英文摘要:

Large language models (LLMs) are increasingly integrated into high-stakes decision-making. Inspired by the theory of \emph{inattentional blindness} in human cognition, we investigate whether LLMs, trained on human-preferred corpora that embed attentional biases, exhibit a similar limitation: \emph{failing to attend to subtle yet important contextual cues under explicit task instructions}. To evaluate this, we introduce the task of \textbf{explicit-implicit reasoning} and present \textbf{MixRea}, a benchmark of 2,246 multiple-choice questions across 9 reasoning types with varying distributions of explicit and implicit information. Evaluation of 21 advanced LLMs shows that even the best-performing reasoning model (Gemini 2.5 Pro) achieves only 42.8\% consistency, revealing widespread inattentional blindness. To mitigate this, we propose \textbf{Potential Relation Completion Prompting (PRCP)}, a prompting method that improves reasoning by recovering overlooked causal relations. Further analysis shows that this limitation persists across diverse multi-source reasoning tasks, highlighting the need for more cognitively aligned models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑