重复提升语言模型嵌入
Repetition Improves Language Model Embeddings
AI总结:
本文提出回声嵌入方法,通过重复输入文本,在不改变架构或微调的情况下,将自回归语言模型转化为高质量嵌入模型,零样本性能提升超5%,并可与监督微调兼容,为统一NLP架构提供新思路。
摘要:
双向模型被认为是强文本嵌入的关键。近期将自回归语言模型(LMs)适配为强文本嵌入模型的方法,大多需要修改LM架构以使其变为双向。我们通过引入“回声嵌入”(echo embeddings)来挑战这一前提,该方法在不改变架构或无需微调的情况下,将自回归LMs转化为高质量的文本嵌入模型。通过重复输入并从重复的标记(这些标记可访问所有原始标记)中提取嵌入,回声嵌入在零样本设置中比经典LM嵌入提升了超过5%。我们的零样本嵌入几乎与经过额外掩码语言建模训练的双向转换LMs所获得的嵌入相匹配。回声嵌入也与监督微调兼容,在苹果对苹果的比较中,即使训练和推理时的计算预算相同,也能匹配或超越双向转换LMs。总体而言,重复是一种简单有效的策略,可规避嵌入模型中对双向注意力的需求,为所有NLP任务走向统一架构铺平道路。
英文摘要:
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive LMs into high quality text embedding models without changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens -- which have access to all original tokens -- echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。