OKBench：通过全自动、按需、开放知识基准测试实现大语言模型评估民主化
OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
AI总结：
针对静态知识基准无法捕捉动态知识、集中式编纂滞后于LLM发展的问题，提出全自动智能体框架OKBench，按需生成新闻领域动态知识基准，评估显示其能揭示模型应对新信息的行为并验证检索对缩小模型性能差距的作用。
摘要：
知识密集型问答是大型语言模型（LLMs）的核心任务，通常使用源自维基百科和教科书等来源的静态基准进行评估。然而，这些基准无法捕捉动态世界中不断演变的知识，且集中式编纂难以跟上LLM的快速发展。为解决这些缺陷，我们提出开放知识基准（OKBench）——一个全自动框架，用于按需生成高质量、动态的知识基准。OKBench聚焦于知识每日更新的新闻领域，是一个自动化基准的来源获取、创建、验证和分发的智能体框架。我们的方法通过减少与预训练数据的重叠，使基准创建民主化，并促进对检索增强方法的全面评估。我们在各种大小和配置的开源及专有LLMs上评估该框架，包括使用和不使用针对新生成知识的检索。结果揭示了模型面对新信息时的不同行为，并强调检索如何缩小小型与大型模型之间的性能差距。这些发现凸显了在演变的知识基准上评估LLMs的重要性。
英文摘要：
Knowledge-intensive question answering is central to large language models (LLMs) and is typically assessed using static benchmarks derived from sources like Wikipedia and textbooks. However, these benchmarks fail to capture evolving knowledge in a dynamic world, and centralized curation struggles to keep pace with rapid LLM advancements. To address these drawbacks, we propose Open Knowledge Bench (OKBench), a fully automated framework for generating high-quality, dynamic knowledge benchmarks on demand. Focusing on the news domain where knowledge updates daily, OKBench is an agentic framework that automates the sourcing, creation, validation, and distribution of benchmarks. Our approach democratizes benchmark creation and facilitates thorough evaluation of retrieval-augmented methods by reducing overlap with pretraining data. We evaluate our framework on a wide range open-source and proprietary LLMs of various sizes and configurations, both with and without retrieval over freshly generated knowledge. Our results reveal distinct model behaviors when confronted with new information and highlight how retrieval narrows the performance gap between small and large models. These findings underscore the importance of evaluating LLMs on evolving knowledge benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
