arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.05928cs.CL
收藏

通过聚焦学习减少长上下文语言模型中的干扰

Reducing Distraction in Long-Context Language Models by Focused Learning

University of Alberta(阿尔伯塔大学);AWS AI Labs(亚马逊云科技人工智能实验室)

Zijun Wu, Bingyuan Liu, Ran Yan, Lei Chen, Thomas Delteil

AI总结:

针对长上下文语言模型中的干扰问题,提出结合检索增强与对比学习的训练方法,提升模型聚焦相关信息的能力,并在长文档问答基准上验证有效性。

摘要:

大型语言模型(LLM)的最新进展显著增强了其处理长上下文的能力。然而,由于干扰问题,有效利用这种长上下文仍然是一个挑战,其中不相关信息主导了长上下文,导致LLM失去对最相关部分的关注。为了解决这个问题,我们提出了一种新颖的训练方法,通过基于检索的数据增强和对比学习的独特组合,增强LLM辨别相关信息的能力。具体来说,在长上下文微调期间,我们使用检索器提取最相关的片段作为增强输入。然后,我们引入一个辅助对比学习目标,明确确保原始上下文和检索到的子上下文的输出紧密对齐。在长单文档和多文档问答基准上的大量实验证明了我们提出的方法的有效性。

英文摘要:

Recent advancements in Large Language Models (LLMs) have significantly enhanced their capacity to process long contexts. However, effectively utilizing this long context remains a challenge due to the issue of distraction, where irrelevant information dominates lengthy contexts, causing LLMs to lose focus on the most relevant segments. To address this, we propose a novel training method that enhances LLMs' ability to discern relevant information through a unique combination of retrieval-based data augmentation and contrastive learning. Specifically, during fine-tuning with long contexts, we employ a retriever to extract the most relevant segments, serving as augmented inputs. We then introduce an auxiliary contrastive learning objective to explicitly ensure that outputs from the original context and the retrieved sub-context are closely aligned. Extensive experiments on long single-document and multi-document QA benchmarks demonstrate the effectiveness of our proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑