arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.18232cs.IR
收藏

长上下文语言模型检索的高效压缩方法

Efficient Long Context Language Model Retrieval with Compression

Minju Seo, Jinheon Baek, Seongyun Lee, Sung Ju Hwang

AI总结:

本文提出专为长上下文语言模型检索设计的段落压缩方法CoLoR,通过合成数据与偏好优化训练,在压缩上下文1.91倍的同时将检索性能提升6%。

摘要:

长上下文语言模型(LCLMs)已成为执行信息检索(IR)的新范式,它通过在单一上下文中处理整个语料库,直接摄取和检索信息,展现出超越传统稀疏和稠密检索方法的潜力。然而,在上下文中处理大量段落进行检索在计算上非常昂贵,并且在推理过程中处理它们的表示会进一步加剧处理时间。因此,我们旨在通过段落压缩使LCLM检索更高效且可能更有效。具体而言,我们提出了一种专为LCLM检索定制的新压缩方法,该方法经过训练,旨在最大化检索性能的同时最小化压缩段落的长度。为此,我们生成了合成数据,其中压缩段落被自动创建,并根据给定查询的检索成功与否标记为被选中或被拒绝。我们利用这些数据通过偏好优化来训练所提出的长上下文检索压缩模型(CoLoR),并在其基础上添加长度正则化损失以强制简洁性。通过在9个数据集上的广泛实验,我们表明CoLoR将检索性能提高了6%,同时将上下文大小压缩了1.91倍。我们的代码可在以下网址获取:https://github.com/going-doer/CoLoR。

英文摘要:

Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR), which enables the direct ingestion and retrieval of information by processing an entire corpus in their single context, showcasing the potential to surpass traditional sparse and dense retrieval methods. However, processing a large number of passages within in-context for retrieval is computationally expensive, and handling their representations during inference further exacerbates the processing time; thus, we aim to make LCLM retrieval more efficient and potentially more effective with passage compression. Specifically, we propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages. To accomplish this, we generate the synthetic data, where compressed passages are automatically created and labeled as chosen or rejected according to their retrieval success for a given query, and we train the proposed Compression model for Long context Retrieval (CoLoR) with this data via preference optimization while adding the length regularization loss on top of it to enforce brevity. Through extensive experiments on 9 datasets, we show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91. Our code is available at: https://github.com/going-doer/CoLoR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑