arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.06679cs.CL
收藏

E2LLM:用于长上下文理解与推理的编码器扩展大语言模型

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

East China Normal University(东华师范大学);Ant Group(蚂蚁集团)

Zihan Liao, Jun Wang, Hang Yu, Lingxiao Wei, Jianguo Li, Jun Wang, Wei Zhang

AI总结:

本文提出E2LLM,通过分块压缩和适配器对齐,解决长上下文处理中的性能、效率与兼容性难题,在文档摘要和问答任务中优于现有方法。

摘要:

处理长上下文对大型语言模型在多轮对话、代码生成和文档摘要等任务中日益重要。本文针对实现高长上下文性能、低计算复杂度和与预训练模型兼容性(统称“不可能三角形”)的挑战,提出E2LLM(编码器扩展大语言模型)。E2LLM将长上下文分成块,使用预训练文本编码器将其压缩为软提示,并通过适配器将这些表示对齐到解码器-only LLM。为增强LLM与这些软提示的推理能力,采用两个训练目标:编码器输出重建和长上下文指令微调。大量实验表明,E2LLM不仅在文档摘要和问答任务中优于8种最先进的方法,在LongBench v2中也达到同等规模模型的最佳性能。

英文摘要:

Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑