arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01372cs.CL
收藏

基于嵌入的上下文提示训练:增强LLM作为文本编码器

Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

Institute of Science Tokyo(东京科学研究院);Tencent(腾讯)

Ailiang Lin, Zhuoyun Li, Keyu Mao, Kotaro Funakoshi, Manabu Okumura

AI总结:

本文提出EPIC方法,通过上下文学习生成高质量嵌入,减少计算负担,实验表明在MTEB基准上取得新突破。

摘要:

大型语言模型(LLMs)已被广泛用于嵌入生成。尽管最近的研究表明,通过在前面添加少量任务相关演示来增强LLM的表示能力,但会导致由于序列长度增加而产生显著的token开销。在本文中,我们提出EPIC,一种新颖的基于嵌入的上下文提示训练策略,利用ICL生成高质量嵌入,同时在训练和推理过程中减少计算负担。该方法用相应连续嵌入替换离散文本演示,不仅鼓励LLM在对比学习中对语义相关的文本对对齐,还要求模型将演示嵌入解释为上下文提示的一部分。因此,EPIC训练的模型在推理时无论是否使用上下文提示都能取得优异的嵌入性能。全面的实验表明,我们的方法在MTEB基准上建立了新的最先进的结果,超越了仅在公开可用检索数据上训练的前沿模型。广泛的消融研究进一步验证了我们机制的有效性和必要性。

英文摘要:

Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑