学习压缩：解锁大型语言模型在文本表示中的潜力
Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
AI总结：
本文提出通过上下文压缩作为预训练任务来提升大型语言模型的文本表示能力，实验表明其在多种任务中优于传统方法。
摘要：
文本表示在聚类、检索和其他下游应用中起着关键作用。随着大型语言模型（LLMs）的出现，越来越多的人关注利用它们的能力进行此目的。然而，大多数LLMs本质上是因果的，并且优化于下一个令牌预测，使其在生成整体表示方面效果不佳。为此，最近的研究引入了预设任务来适应LLMs进行文本表示。然而，大多数这些任务却依赖于令牌级预测目标，如在LLM2Vec中使用的掩码下一个令牌预测（MNTP）。在本工作中，我们探索了上下文压缩作为预训练任务来无监督适应LLMs的未被开发的潜力。在压缩预训练过程中，模型学习生成紧凑的记忆令牌，这些令牌替代整个上下文用于下游序列预测。实验表明，一个精心设计的压缩目标可以显著增强基于LLM的文本表示，优于使用令牌级预设任务训练的模型。通过对比学习进一步改进，产生了一个强大的表示模型（LLM2Comp），在广泛的任务上优于当前的基于LLM的文本编码器，同时更高效，需要显著较少的训练数据。代码可在https://github.com/longtaizi13579/LLM2Comp上获得。
英文摘要：
Text representation plays a critical role in tasks like clustering, retrieval, and other downstream applications. With the emergence of large language models (LLMs), there is increasing interest in harnessing their capabilities for this purpose. However, most of the LLMs are inherently causal and optimized for next-token prediction, making them suboptimal for producing holistic representations. To address this, recent studies introduced pretext tasks to adapt LLMs for text representation. Most of these tasks, however, rely on token-level prediction objectives, such as the masked next-token prediction (MNTP) used in LLM2Vec. In this work, we explore the untapped potential of context compression as a pretext task for unsupervised adaptation of LLMs. During compression pre-training, the model learns to generate compact memory tokens, which substitute the whole context for downstream sequence prediction. Experiments demonstrate that a well-designed compression objective can significantly enhance LLM-based text representations, outperforming models trained with token-level pretext tasks. Further improvements through contrastive learning produce a strong representation model (LLM2Comp) that outperforms contemporary LLM-based text encoders on a wide range of tasks while being more sample-efficient, requiring significantly less training data. Code is available at https://github.com/longtaizi13579/LLM2Comp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
