Encyclo-K：通过动态组成的知识陈述评估LLM
Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
AI总结：
Encyclo-K通过动态组合知识陈述，提出了一种评估LLM全面理解能力的基准测试框架，有效解决了现有基准测试的局限性。
摘要：
基准测试在追踪大型语言模型（LLM）的快速进步和确定其能力边界方面起着至关重要的作用。然而，现有的基准测试主要在问题层面进行问题的收集，存在三个根本性的局限性：易受数据污染、局限于单个知识点评估以及依赖于昂贵的领域专家注释。我们提出了Encyclo-K，一个基于陈述的基准测试，从基础重新思考基准测试的构建。我们的关键见解是，知识陈述，而不是问题，可以作为收集的单位，然后可以从它们中构建问题。我们从权威教科书中提取独立的知识陈述，并通过在测试时进行随机抽样动态组成评估问题。这种设计直接解决了这三个限制：组合空间太大无法记忆，且模型排名在动态生成的问题集中保持稳定，使能够可靠地定期刷新数据集；每个问题汇总8-10个陈述，以实现全面的多知识评估；注释员只需验证格式合规性，而不需要领域专业知识，大大降低了注释成本。在超过50个LLM上的实验表明，Encyclo-K提出了具有强大区分能力的挑战。即使表现最好的OpenAI-GPT-5.1也只达到了62.07%的准确率，且模型性能呈现出明显的梯度分布——推理模型从16.04%到62.07%，而聊天模型从9.71%到50.40%。这些结果验证了动态评估和多陈述全面理解所带来的挑战。这些发现确立了Encyclo-K作为动态评估LLM在多个细粒度学科知识陈述上的全面理解的可扩展框架。
英文摘要：
Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predominantly curate questions at the question level, suffering from three fundamental limitations: vulnerability to data contamination, restriction to single-knowledge-point assessment, and reliance on costly domain expert annotation. We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up. Our key insight is that knowledge statements, not questions, can serve as the unit of curation, and questions can then be constructed from them. We extract standalone knowledge statements from authoritative textbooks and dynamically compose them into evaluation questions through random sampling at test time. This design directly addresses all three limitations: the combinatorial space is too vast to memorize, and model rankings remain stable across dynamically generated question sets, enabling reliable periodic dataset refresh; each question aggregates 8-10 statements for comprehensive multi-knowledge assessment; annotators only verify formatting compliance without requiring domain expertise, substantially reducing annotation costs. Experiments on over 50 LLMs demonstrate that Encyclo-K poses substantial challenges with strong discriminative power. Even the top-performing OpenAI-GPT-5.1 achieves only 62.07% accuracy, and model performance displays a clear gradient distribution--reasoning models span from 16.04% to 62.07%, while chat models range from 9.71% to 50.40%. These results validate the challenges introduced by dynamic evaluation and multi-statement comprehensive understanding. These findings establish Encyclo-K as a scalable framework for dynamic evaluation of LLMs' comprehensive understanding over multiple fine-grained disciplinary knowledge statements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
