面向句子语义表示的静态词嵌入
Static Word Embeddings for Sentence Semantic Representation
AI总结：
本研究提出优化用于句子语义表示的新型静态词嵌入，从预训练句子Transformer提取词嵌入后经句子级主成分分析、知识蒸馏或对比学习改进，推理时取平均即可得句子表示，性能优于现有静态模型甚至基础SimCSE模型，还可移除无关成分并调整向量范数。
摘要：
我们提出了专为句子语义表示优化的新型静态词嵌入。首先从预训练的Sentence Transformer（句子Transformer）中提取词嵌入，通过句子级主成分分析进行优化，随后采用知识蒸馏或对比学习进一步改进。推理阶段仅需对词嵌入取平均即可得到句子表示，计算成本极低。我们在单语和跨语任务上评估了模型，结果显示该模型在句子语义任务上大幅优于现有静态模型，在文本嵌入基准上甚至超越了基础Sentence Transformer模型SimCSE。最后通过多项分析表明，该方法成功移除了词嵌入中与句子语义相关性不高的成分，并根据词语对句子语义的影响调整了向量范数。
英文摘要：
We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are not highly relevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
