arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.20535cs.CLcs.AI
收藏

ARC-Encoder:为大语言模型学习压缩文本表示

ARC-Encoder: learning compressed text representations for large language models

Kyutai

Hippolyte Pilchen, Edouard Grave, Patrick Pérez

AI总结:

ARC-Encoder是可将文本压缩为原词元数1/4或1/8的编码器,适配多类LLM解码器,在提升推理效率的同时保持最优性能,为大语言模型提供灵活高效的上下文压缩方案。

摘要:

检索增强生成、思维链推理等技术已使大语言模型(LLM)的上下文长度增加,但也提升了推理成本。上下文压缩技术可降低该成本,不过多数有效方法需对目标模型进行微调甚至修改其架构,这会导致模型在非特定用途场景下的通用能力下降。本文探索了一种替代方案:将上下文压缩为连续表示的编码器,用于替换解码器LLM中的词元嵌入。首先,我们系统研究了编码器的训练策略与架构选择,据此设计出自适应文本表示压缩器ARC-Encoder,其输出的连续表示数量仅为文本词元的1/x(通常x∈{4,8})。我们在多种LLM使用场景(从上下文学习到上下文窗口扩展)、指令型及基础型解码器上对ARC-Encoder进行评估,结果显示其在多个基准测试中达到了当前最优性能,同时提升了推理计算效率。最后,我们证明该模型可同时适配多个解码器,使单个编码器能跨不同解码器LLM泛化,成为可与多种LLM无缝协作的灵活高效便携式编码器方案。训练代码、微调数据集及预训练模型均已发布。

英文摘要:

Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑