arXiv 2506.231282025-06-29cs.AI

大型语言模型是否具备深度关系推理能力？来自DeepSeek-R1的见解与基准比较

Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons

School of Professional Education and Executive Development（专业教育与执行发展学院）；The Hong Kong Polytechnic University（香港理工大学）；School of Mathematics and Statistics（数学与统计学学院）；Beijing Institute of Technology（北京理工大学）；Department of Mathematics（数学系）；The University of Hong Kong（香港大学）

Chi Chiu So, Yueyue Sun, Jun-Min Wang, Siu Pang Yung, Anthony Wai Keung Loh, Chun Pong Chau

备注：: 10 pages, 0 figures, accepted by 2025 IEEE international conference on artificial intelligence testing (AITest)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过家谱和通用图推理基准任务评估了DeepSeek-R1等三种前沿LLMs的深度关系推理能力，发现DeepSeek-R1表现最佳，但所有模型在复杂问题上受限于token长度和输出结构，并揭示了其推理策略与不足。

摘要：

大型语言模型（LLMs）在执行深度关系推理方面表现如何？本文通过一系列精心设计的家谱和通用图推理基准任务，评估并比较了DeepSeek-R1、DeepSeek-V3和GPT-4o三种前沿LLMs的推理能力。实验表明，DeepSeek-R1在多种任务和问题规模下均持续取得最高的F1分数，展现出强大的逻辑演绎和关系推理天赋。然而，随着问题复杂度增加，所有评估模型（包括DeepSeek-R1）都明显遇到困难，主要归因于token长度限制和不完整的输出结构。对DeepSeek-R1的长思维链响应的详细分析揭示了其独特的规划和验证策略，但也凸显了推理不连贯或不完整的实例，呼吁对LLMs的内部推理动态进行更深入的审查。我们进一步讨论了未来工作的关键方向，包括多模态推理的作用及对推理失败的系统性检验。我们的发现为提升LLMs的推理能力提供了实证见解和理论启示，特别是在需要结构化、多步逻辑推理的任务中。我们的代码仓库将公开于https://github.com/kelvinhkcs/Deep-Relational-Reasoning。

英文摘要：

How far are Large Language Models (LLMs) in performing deep relational reasoning? In this paper, we evaluate and compare the reasoning capabilities of three cutting-edge LLMs, namely, DeepSeek-R1, DeepSeek-V3 and GPT-4o, through a suite of carefully designed benchmark tasks in family tree and general graph reasoning. Our experiments reveal that DeepSeek-R1 consistently achieves the highest F1-scores across multiple tasks and problem sizes, demonstrating strong aptitude in logical deduction and relational inference. However, all evaluated models, including DeepSeek-R1, struggle significantly as problem complexity increases, largely due to token length limitations and incomplete output structures. A detailed analysis of DeepSeek-R1's long Chain-of-Thought responses uncovers its unique planning and verification strategies, but also highlights instances of incoherent or incomplete reasoning, calling attention to the need for deeper scrutiny into LLMs' internal inference dynamics. We further discuss key directions for future work, including the role of multimodal reasoning and the systematic examination of reasoning failures. Our findings provide both empirical insights and theoretical implications for advancing LLMs' reasoning abilities, particularly in tasks that demand structured, multi-step logical inference. Our code repository will be publicly available at https://github.com/kelvinhkcs/Deep-Relational-Reasoning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。