arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.24932cs.CL
收藏

RiddleBench:面向大语言模型的新型生成式推理基准

RiddleBench: A New Generative Reasoning Benchmark for LLMs

Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);IT University of Copenhagen(哥本哈根技术大学);Microsoft(微软);Indian Institute of Engineering, Science and Technology, Shibpur(Shibpur印度工程科学技术学院)

Deepon Halder, Alan Saji, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan, Raj Dabre

AI总结:

针对现有推理基准难以评估灵活多面推理能力的问题,该研究提出包含1737道英文谜题的RiddleBench基准,发现前沿大模型准确率仅略超60%,存在幻觉级联、自我纠错差、推理脆弱等问题,可为模型研发提供诊断和指导。

摘要:

大语言模型在诸多成熟推理基准上已展现出强劲性能。然而,这些基准主要评估定量问题求解等结构化技能,在评估人类智能核心的灵活、多方面推理能力方面存在空白。这类能力需要将逻辑演绎与空间感知、约束满足相结合,而当前的评估方式无法很好地衡量这些能力。为解决这一问题,我们推出了RiddleBench——一个包含1737道高难度英文谜题的基准,旨在探测这些核心推理能力。在RiddleBench上对前沿模型的评估暴露了根本性缺陷。即便是Gemini 2.5 Pro、o3和Claude 4 Sonnet这类顶级专有模型,准确率也仅略高于60%(分别为60.30%、63.37%和63.16%)。进一步分析揭示了深层问题,包括幻觉级联(采信其他模型的错误推理)以及因强烈的自我确认偏差导致的自我纠错能力薄弱。这些模型的推理还十分脆弱:当约束条件被重新排序或引入无关信息时,性能会显著下降。RiddleBench可作为诊断上述问题的工具,也能作为资源指导开发更稳健、可靠的语言模型。

英文摘要:

Large Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑