AI 中文总结
本文提出推理时自对比引导（SCS）方法，通过构建无条件嵌入并干预注意力机制，提升大语言模型的条件文本嵌入质量，无需训练且即插即用。
AI 中文摘要
从大语言模型（LLMs）中提取条件文本嵌入是一种有前景的范式，因为它既不需要额外数据，也不需要微调。现有方法将条件融入提示中，以引导LLMs关注特定方面并生成条件文本嵌入。然而，仅依赖提示往往无法产生高质量的条件文本嵌入，因为它们仍与通用文本嵌入纠缠在一起，最终降低了其质量。为此，我们提出了一种推理时的即插即用自对比引导（SCS）方法，该方法构建无条件通用文本嵌入，并利用它们来优化条件文本嵌入，使其更聚焦于目标条件。具体而言，我们修改注意力掩码和位置编码以遮蔽条件，从而获得无条件文本嵌入，并干预多头自注意力计算过程。值得注意的是，我们的方法非常高效，在推理时仅需一次额外的多头自注意力计算。在聚类、语义文本相似性和三元组对齐数据集上的大量实验表明，我们的方法能够以无需训练且即插即用的方式，无缝提升现有基于提示的方法在不同LLM上的性能。我们的代码将在此https URL发布。
英文摘要
Extracting conditional text embeddings from large language models (LLMs) is a promising paradigm, as it requires neither additional data nor fine-tuning. Existing methods incorporate conditions into prompts to guide LLMs to focus on specific aspects and elicit conditional text embeddings. However, relying solely on prompts often fails to produce high-quality conditional text embeddings, as they remain entangled with general text embeddings, ultimately degrading their quality. To this end, we propose an inference-time, plug-and-play Self-Contrastive Steering (SCS) method that constructs unconditional general text embeddings and uses them to refine conditional text embeddings, making them more focused on the target condition. Specifically, we modify the attention mask and positional encodings to mask the condition, thereby obtaining unconditional text embeddings and intervening in the multi-head self-attention computation process. Notably, our method is highly efficient, requiring only a single additional multi-head self-attention computation at inference time. Extensive experiments on clustering, Semantic Textual Similarity, and triplet alignment datasets demonstrate that our method can seamlessly improve the performance of existing prompt-based methods across different LLMs in a training-free and plug-and-play manner. Our code will be released at https://github.com/zifengcheng/SCS
CommentsACL 2026 (Oral)
