KnowledgeBerg: 评估大语言模型中的系统性知识覆盖与组合推理
KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models
AI总结：
提出KnowledgeBerg基准，通过4800道选择题评估大模型在知识宽度和推理深度上的系统性覆盖与组合推理能力，发现现有模型存在严重不足。
摘要：
许多现实世界的问题看似简单，却隐含地要求两种能力：(i) 对有限知识宇宙的系统性覆盖，以及(ii) 对该宇宙的基于集合的组合推理，我们将这种现象称为“冰山一角”。我们通过两个正交维度形式化这一挑战：知识宽度（所需宇宙的基数）和推理深度（组合集合操作的数量）。我们引入了KnowledgeBerg，一个包含4800道多项选择题的基准，这些题目源自1183个枚举种子，涵盖10个领域和17种语言，其宇宙基于权威来源以确保可重复性。代表性的开源大语言模型表现出严重局限性，在宇宙枚举上仅达到5.26-36.88的F1分数，在基于知识的推理上准确率仅为16.00-44.19。诊断分析揭示了三个失败阶段：完整性（知识缺失）、意识（未能识别需求）和应用（错误执行推理）。这种模式在语言和模型规模上持续存在。尽管测试时计算和检索增强带来了可测量的改进——分别高达4.35和3.78个百分点——但仍有显著差距，暴露了当前大语言模型在组织结构化知识和在有限领域上执行组合推理方面的局限性。数据集可在https://huggingface.co/datasets/2npc/KnowledgeBerg获取。
英文摘要：
Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg." We formalize this challenge through two orthogonal dimensions: knowledge width, the cardinality of the required universe, and reasoning depth, the number of compositional set operations. We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions derived from 1,183 enumeration seeds spanning 10 domains and 17 languages, with universes grounded in authoritative sources to ensure reproducibility. Representative open-source LLMs demonstrate severe limitations, achieving only 5.26-36.88 F1 on universe enumeration and 16.00-44.19 accuracy on knowledge-grounded reasoning. Diagnostic analyses reveal three stages of failure: completeness, or missing knowledge; awareness, or failure to identify requirements; and application, or incorrect reasoning execution. This pattern persists across languages and model scales. Although test-time compute and retrieval augmentation yield measurable gains -- up to 4.35 and 3.78 points, respectively -- substantial gaps remain, exposing limitations in how current LLMs organize structured knowledge and execute compositional reasoning over bounded domains. The dataset is available at https://huggingface.co/datasets/2npc/KnowledgeBerg
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
