NeedleBench:评估LLM在不同信息密度下的检索与推理能力
NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
AI总结:
本文提出NeedleBench框架,通过在不同深度嵌入关键数据点,评估LLM在信息稀疏与密集场景下的双语长上下文检索与推理能力,揭示了最新推理模型在连续检索推理中存在的“思考不足”现象。
摘要:
大型语言模型处理长上下文信息的能力在各种实际应用中至关重要。现有的评估方法通常要么依赖真实世界的长文本,这使得难以排除模型固有知识的影响;要么引入无关的填充内容以人为达到目标长度,从而降低了评估效果。为了解决这些局限性,我们引入了 NeedleBench,这是一个用于评估具有自适应上下文长度的双语长上下文任务中检索和推理性能的合成框架。NeedleBench 系统地在不同深度嵌入关键数据点,以严格测试模型能力。任务被划分为两种场景:信息稀疏场景,在大量无关文本中包含极少的相关细节,以模拟简单的检索任务;以及信息密集场景(Ancestral Trace Challenge),相关信息连续分布在整个上下文中,以模拟复杂的推理任务。我们的实验表明,尽管像 Deepseek-R1 和 OpenAI 的 o3 这样的最新推理模型在数学推理方面表现出色,但它们在信息密集场景中即使上下文长度较短也难以进行连续的检索和推理。我们还描述了一种称为“思考不足(under-thinking)”的现象,即模型尽管有可用信息却过早得出推理结论。因此,NeedleBench 提供了关键的见解和有针对性的工具,这对于评估和改进 LLM 的长上下文能力至关重要。所有资源可在 OpenCompass 获取:https://github.com/open-compass/opencompass。
英文摘要:
The capability of large language models to handle long-context information is crucial across various real-world applications. Existing evaluation methods often rely either on real-world long texts, making it difficult to exclude the influence of models' inherent knowledge, or introduce irrelevant filler content to artificially achieve target lengths, reducing assessment effectiveness. To address these limitations, we introduce NeedleBench, a synthetic framework for assessing retrieval and reasoning performance in bilingual long-context tasks with adaptive context lengths. NeedleBench systematically embeds key data points at varying depths to rigorously test model capabilities. Tasks are categorized into two scenarios: information-sparse, featuring minimal relevant details within extensive irrelevant text to simulate simple retrieval tasks; and information-dense (the Ancestral Trace Challenge), where relevant information is continuously distributed throughout the context to simulate complex reasoning tasks. Our experiments reveal that although recent reasoning models like Deepseek-R1 and OpenAI's o3 excel in mathematical reasoning, they struggle with continuous retrieval and reasoning in information-dense scenarios, even at shorter context lengths. We also characterize a phenomenon termed 'under-thinking', where models prematurely conclude reasoning despite available information. NeedleBench thus provides critical insights and targeted tools essential for evaluating and improving LLMs' long-context capabilities. All resources are available at OpenCompass: https://github.com/open-compass/opencompass.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。