arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.08082cs.CLcs.AI
收藏

短上下文主导:自然语言实际上需要多少局部上下文?

Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

University of British Columbia(不列颠哥伦比亚大学);Google DeepMind(谷歌DeepMind)

Vala Vakilian, Zimeng Wang, Ankit Singh Rawat, Christos Thrampoulidis

AI总结:

研究探讨了自然语言中短上下文预测的主导地位,并提出DaMCL方法用于检测长上下文序列,通过减少偏差提升模型性能。

摘要:

我们研究了短上下文主导假设:对于大多数序列,一小部分局部前缀就足以预测其下一个标记。使用大型语言模型作为统计预言机,我们测量了在不同长度的序列数据集上重现准确全上下文预测所需的最小上下文长度(MCL)。对于来自长上下文文档的1-7k个标记的序列,我们一致发现,75-80%的序列仅需要最多最后96个标记即可。鉴于短上下文标记的主导地位,我们随后问是否有可能检测出具有挑战性的长上下文序列,其中短局部前缀不足以进行预测。我们引入了一个实用的代理,称为分布感知MCL(DaMCL),它不需要实际下一个标记的知识,并且与超越贪婪解码的采样策略兼容。我们的实验验证了简单阈值化DaMCL定义的指标在检测长 vs 短上下文序列方面的高性能。最后,为了对抗短上下文主导在LLM输出分布中引入的偏差,我们开发了一种直观的解码算法,利用我们的检测器来识别并增强长距离相关的标记。在问答任务和模型架构上,我们确认通过减轻偏差可以提高性能。

英文摘要:

We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistical oracles, we measure the minimum context length (MCL) needed to reproduce accurate full-context predictions across datasets with sequences of varying lengths. For sequences with 1-7k tokens from long-context documents, we consistently find that 75-80% require only the last 96 tokens at most. Given the dominance of short-context tokens, we then ask whether it is possible to detect challenging long-context sequences for which a short local prefix does not suffice for prediction. We introduce a practical proxy to MCL, called Distributionally Aware MCL (DaMCL), that does not require knowledge of the actual next-token and is compatible with sampling strategies beyond greedy decoding. Our experiments validate that simple thresholding of the metric defining DaMCL achieves high performance in detecting long vs. short context sequences. Finally, to counter the bias that short-context dominance induces in LLM output distributions, we develop an intuitive decoding algorithm that leverages our detector to identify and boost tokens that are long-range-relevant. Across Q&A tasks and model architectures, we confirm that mitigating the bias improves performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑