CL-bench: 一个用于上下文学习的基准测试
CL-bench: A Benchmark for Context Learning
AI总结:
CL-bench是一个用于评估上下文学习能力的基准测试,通过复杂任务和上下文设计,揭示当前语言模型在真实任务中的学习瓶颈。
摘要:
当前的语言模型(LMs)在使用预训练知识进行提示推理方面表现出色。然而,现实中的任务远比这更复杂且依赖于上下文:模型必须从特定任务的上下文中学习,并利用超出预训练阶段所学的新知识来推理和解决任务。我们将这种能力称为上下文学习,这是人类自然具备但长期以来被忽视的能力。为此,我们引入了CL-bench,一个由500个复杂上下文、1,899个任务和31,607个验证标准组成的现实世界基准测试,均由经验丰富的领域专家精心设计。每个任务都设计成所需的新内容包含在对应的上下文中。在CL-bench中解决任务需要模型从上下文中学习,包括新的领域特定知识、规则系统、复杂程序以及从实证数据中推导出的法律,这些内容在预训练阶段都不存在。这远远超出了长上下文任务,这些任务主要测试检索或阅读理解,以及上下文学习任务,其中模型通过指令和演示学习简单的任务模式。我们对十种前沿LMs的评估发现,模型平均只能解决17.2%的任务。即使表现最好的模型GPT-5.1也只能解决23.7%,这表明LMs尚未实现有效的上下文学习,这成为解决现实世界复杂上下文依赖任务的关键瓶颈。CL-bench代表了朝着构建具备这种基本能力的LMs迈出的一步,使它们更加智能,并推进其在现实世界中的应用。
英文摘要:
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。