基于对比学习的自适应重构无监督句嵌入方法
Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings
AI总结：
针对预训练语言模型词元偏置导致无监督句嵌入细粒度语义丢失的问题，提出SARCSE框架，通过自编码器重构全部词元并引入自适应重构损失，在7个STS任务上显著优于SimCSE。
摘要：
无监督句嵌入任务旨在将句子转换为语义向量表示。以往大多数工作直接使用从预训练语言模型中获得的句子表示。然而，由于预训练语言模型中的词元偏置，模型无法捕捉句子中的细粒度语义，从而导致预测效果不佳。为解决这一问题，我们提出了一种新颖的自适应重构对比句嵌入（SARCSE）框架，该框架利用自编码器重构句子中的所有词元，以帮助模型在词元聚合过程中保留更多细粒度语义。此外，我们提出了一种自适应重构损失，以缓解词元对频率的偏置。实验结果表明，与强基线SimCSE相比，SARCSE在7个STS任务上取得了显著提升。
英文摘要：
Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
