arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.09571cs.CLcs.ITmath.IT
收藏

通过语义压缩扩展大语言模型的上下文窗口

Extending Context Window of Large Language Models via Semantic Compression

Tsinghua University(清华大学);Huawei Technologies Co., Ltd.(华为技术有限公司)

Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, Wei Han

AI总结:

针对大语言模型上下文长度受限的问题,提出一种基于信息论信源编码、无需微调的语义压缩方法,可将上下文扩展6-8倍,在多类任务中有效且能降低计算开销、保持生成流畅度。

摘要:

基于Transformer的大语言模型(LLM)通常会对输入文本长度施加限制,以确保生成流畅且相关的响应。这一约束限制了它们在涉及长文本场景中的适用性。我们提出了一种新颖的语义压缩方法,无需承担显著计算成本也无需微调,就能将模型泛化能力扩展至6-8倍长度的文本。我们提出的框架受信息论中信源编码的启发,采用预训练模型在将长输入传递给LLM用于下游任务之前,降低其语义冗余。实验结果表明,我们的方法在问答、摘要生成、少样本学习和信息检索等一系列任务中,有效扩展了LLM的上下文窗口。此外,所提出的语义压缩方法在降低相关计算开销的同时,还能保持文本生成的流畅性一致。

英文摘要:

Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer, without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑