arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.13626cs.CLcs.AI
收藏

LIFT:通过长输入微调提升长上下文理解能力

LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(BIGAI通用人工智能全国重点实验室)

Yansheng Mao, Jiaqi Li, Fanxu Meng, Jing Xiong, Zilong Zheng, Muhan Zhang

AI总结:

提出 LIFT 框架,通过测试时调整模型参数适应上下文,使短上下文模型高效处理长输入,结合上下文学习在长上下文基准上显著提升性能。

摘要:

由于上下文窗口受限,长上下文理解对大语言模型仍具挑战。本文提出用于长上下文建模的 Long Input Fine-Tuning (LIFT),这是一种通过在测试时调整模型参数以适应上下文,从而增强 LLM 在长上下文任务上性能的新框架。LIFT 能高效处理长输入,免除了离线长上下文适应的计算负担,并能提升任意短上下文模型的长上下文能力。该框架通过整合上下文学习和 pre-LIFT 监督微调得到进一步增强。上下文学习与 LIFT 的结合使 Llama 3 等短上下文模型能处理任意长上下文,并在 LooGLE 和 LongBench 等流行长上下文基准上持续提升性能。我们还全面分析了 LIFT 在长上下文理解上的优势与局限,为未来研究提供了宝贵方向。

英文摘要:

Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑