arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22067cs.CL
收藏

超越上下文长度:表征长上下文推理的难度

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

University of Edinburgh(爱丁堡大学)

Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li

AI总结:

该研究针对长上下文推理任务,提出用证据范围、分散度和推理深度补充上下文长度来表征难度,构建了LURCH基准并验证了相关属性与模型性能的关系。

摘要:

长上下文语言模型支持越来越长的上下文窗口,而评估通常将上下文长度视为难度的主要指标。然而,仅上下文长度无法解释模型为何在需要对复杂信息进行推理的长上下文任务上失败。相同长度的上下文可能带来截然不同的挑战:一项任务可能可从单个局部段落中解决,也可能需要整合分布在整个上下文的证据。我们通过三个属性定量表征这种难度:证据范围(需查找的必要信息量)、证据分散度(该信息在上下文内的分散程度)以及推理深度(从证据中推导答案所需的依赖步骤数)。我们构建了LURCH,这是一个包含6550个实例的多领域问答基准,带有支持证据和推理图,可对所有三个属性进行量化。对专有模型和开放权重模型的评估显示:(1)在固定上下文长度下,我们的所有三个属性均与模型性能呈强负相关;(2)随着上下文长度增加,证据分散的样本性能急剧下降,而证据局部化的样本性能相对稳定;(3)难度在属性间复合,三个属性均处于中等较高值的问题比仅一个属性处于极端值的问题更难。这些发现确立了证据范围、分散度和推理深度作为长上下文推理难度的衡量指标,可补充上下文长度的不足。

英文摘要:

Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑