NeedleChain: 测量大型语言模型的完整上下文理解能力
NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models
AI总结：
针对现有基准高估LLM上下文理解能力的问题，提出NeedleChain基准，通过全相关上下文测试模型整合所有证据的能力，并引入ROPE收缩策略提升全上下文整合。
摘要：
最近的报告表明，LLM可以处理越来越长的上下文。然而，许多现有的上下文理解基准嵌入了大量与查询无关的内容，这使评估转向检索相关片段，而非完全整合所有提供的信息。在这种设置下，我们认为当前基准可能高估了LLM真正的上下文理解能力。特别是，我们证明当上下文完全由与查询相关的文本组成时，即使是GPT-4o这样的先进模型也无法可靠地整合短至200个token的输入。为了更严格地评估这种能力，我们引入了NeedleChain，一个旨在测试模型是否能忠实地整合所有给定证据的基准。NeedleChain包括三种变体，它们在理解所需顺序上有所不同，以及一个基于大海捞针（NIAH）范式的并行基准。通过比较这些变体，NeedleChain能够更全面地评估上下文理解。我们进一步提出了一种无需训练的策略，鼓励模型反映所有可用信息，即ROPE收缩，强调了全上下文整合的重要性，并为改进上下文可靠推理指明了新方向。
英文摘要：
Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether models can faithfully incorporate all given evidence. NeedleChain includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-in-a-haystack(NIAH) paradigm. By comparing these variants, NeedleChain enables a more comprehensive assessment of context understanding. We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
