arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.05862cs.CLcs.AI
收藏

从上下文去噪视角重新审视长上下文建模

Revisiting Long-context Modeling from Context Denoising Perspective

Soochow University(苏州大学);LCM Laboratory(长文实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Zecheng Tang, Baibei Ji, Juntao Li, Lijun Wu, Haijia Gui, Min Zhang

AI总结:

本文从上下文去噪视角分析长上下文模型,提出IG分数量化噪声,并设计CDT训练策略以增强模型对关键标记的注意力,使8B模型性能媲美GPT-4o。

摘要:

长上下文模型(LCMs)在处理长序列方面展现出了巨大潜力,促进了许多实际应用。LCMs的成功归功于其能够在上下文中定位隐式关键信息以进行进一步预测的能力。然而,最近的研究表明,LCMs通常容易受到上下文噪声(即无关标记)的影响,这些噪声会误导模型注意力。在本文中,我们对上下文噪声进行了细粒度分析,并提出了一种有效的指标——Integrated Gradient(IG)分数,用于检测和量化上下文中的噪声信息。我们的发现表明,即使是对检测到的上下文噪声进行简单缓解,也能大幅提升模型对关键标记的注意力,并有利于后续预测。基于这一见解,我们提出了Context Denoising Training(CDT),这是一种简单而有效的训练策略,能够提升对关键标记的注意力,同时强化它们对模型预测的影响。在上下文窗口缩放和长上下文对齐两种设置下,跨四个任务的广泛实验证明了CDT的优越性。值得注意的是,使用CDT训练时,一个开源的8B模型能够取得(50.92)与GPT-4o(51.00)相当的性能。

英文摘要:

Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their ability to locate implicit critical information within the context for further prediction. However, recent research reveals that LCMs are often susceptible to contextual noise, i.e., irrelevant tokens, that can mislead model attention. In this paper, we conduct a fine-grained analysis of the context noise and propose an effective metric, the Integrated Gradient (IG) score, to detect and quantify the noise information within the context. Our findings reveal that even simple mitigation of detected context noise can substantially boost the model's attention on critical tokens and benefit subsequent predictions. Building on this insight, we propose Context Denoising Training (CDT), a straightforward yet effective training strategy that improves attention on critical tokens while reinforcing their influence on model predictions. Extensive experiments across four tasks, under both context window scaling and long-context alignment settings, demonstrate the superiority of CDT. Notably, when trained with CDT, an open-source 8B model can achieve performance (50.92) comparable to GPT-4o (51.00).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑