CLR-Fact：评估大型语言模型在事实知识上的复杂逻辑推理能力
CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge
AI总结：
本文提出CLR-Fact基准，系统评估LLMs在通用与生物医学知识图谱上的复杂逻辑推理能力，发现思维链提示可显著提升性能，且模型擅长并集运算但交集推理薄弱。
摘要：
虽然大型语言模型（LLMs）通过从广泛的训练数据中获取丰富的事实知识，已在多种自然语言处理任务中展现出令人印象深刻的能力，但它们在复杂方式下综合并逻辑推理这些知识的能力仍未得到充分探索。在本工作中，我们通过一个基于通用领域和生物医学知识图谱自动生成的复杂推理问题新基准，对最先进LLMs的复杂逻辑推理能力进行了系统性评估。我们采用多样化的上下文学习技术开展大量实验，结果表明LLMs擅长对通用世界知识进行推理，但在专业领域特定知识上面临显著挑战。我们发现，使用显式的思维链（Chain-of-Thought）示例进行提示，可以大幅提升LLMs在包含多种逻辑运算的复杂逻辑推理任务上的表现。有趣的是，我们的受控评估揭示了一种不对称现象：LLMs在集合并集运算上表现出熟练能力，但在集合交集运算上却相当吃力——而交集是逻辑推理的关键构建块。为促进后续研究，我们将公开发布评估基准和代码。
英文摘要：
While large language models (LLMs) have demonstrated impressive capabilities across various natural language processing tasks by acquiring rich factual knowledge from their broad training data, their ability to synthesize and logically reason with this knowledge in complex ways remains underexplored. In this work, we present a systematic evaluation of state-of-the-art LLMs' complex logical reasoning abilities through a novel benchmark of automatically generated complex reasoning questions over general domain and biomedical knowledge graphs. Our extensive experiments, employing diverse in-context learning techniques, reveal that LLMs excel at reasoning over general world knowledge but face significant challenges with specialized domain-specific knowledge. We find that prompting with explicit Chain-of-Thought demonstrations can substantially improve LLM performance on complex logical reasoning tasks with diverse logical operations. Interestingly, our controlled evaluations uncover an asymmetry where LLMs display proficiency at set union operations, but struggle considerably with set intersections - a key building block of logical reasoning. To foster further work, we will publicly release our evaluation benchmark and code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
