以多跳推理视角基准测试中文常识推理
Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
AI总结：
提出中文常识多跳推理基准CCMOR，通过LLM流水线生成多跳问题并人工验证，评估发现大模型在长尾知识与知识密集型推理上存在局限，而检索增强生成可显著提升性能。
摘要：
尽管大型语言模型（LLMs）已展现出先进的推理能力，但对其在通用中文语境中的全面评估仍研究不足。为弥补这一空白，我们提出了中文常识多跳推理基准（CCMOR），这是一个新颖的基准，旨在评估LLMs将中文特有的事实知识与多步逻辑推理相结合的能力。具体而言，我们首先从现有问答数据集中构建一个领域均衡的种子集，然后开发一个基于LLM的流水线，以生成锚定在事实单元链上的多跳问题。为确保所得数据集的质量，我们实施了一个人在回路中的验证系统，由领域专家系统性地验证并改进生成的问题。利用CCMOR，我们评估了最先进的LLMs，展示了LLMs在处理长尾知识和执行知识密集型推理方面持续存在的局限性。值得注意的是，检索增强生成显著缓解了这些知识差距，带来了显著的性能提升。
英文摘要：
While Large Language Models (LLMs) have demonstrated advanced reasoning capabilities, their comprehensive evaluation in general Chinese-language contexts remains understudied. To bridge this gap, we propose Chinese Commonsense Multi-hop Reasoning (CCMOR), a novel benchmark designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning. Specifically, we first construct a domain-balanced seed set from existing QA datasets, then develop an LLM-powered pipeline to generate multi-hop questions anchored on factual unit chains. To ensure the quality of resulting dataset, we implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions. Using CCMOR, we evaluate state-of-the-art LLMs, demonstrating persistent limitations in LLMs' ability to process long-tail knowledge and execute knowledge-intensive reasoning. Notably, retrieval-augmented generation substantially mitigates these knowledge gaps, yielding significant performance gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
