arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.06154cs.CL
收藏

缓解无监督句子表示中的过度平滑问题

Alleviating Over-smoothing for Unsupervised Sentence Representation

Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);Microsoft(微软);Duke University(杜克大学);Peking University(北京大学)

Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Bowen Cao, Jianhui Chang, Daxin Jiang, Jia Li

AI总结:

针对无监督句子表示中的过度平滑问题,提出自对比学习(SSCL)方法,从预训练模型中间层采样负样本,提升表示质量,并在多个基线上取得显著性能提升。

摘要:

目前,学习更好的无监督句子表示是许多自然语言处理社区追求的目标。基于预训练语言模型(PLMs)和对比学习的许多方法在该任务上已取得有前景的结果。通过实验,我们观察到过度平滑问题降低了这些强大PLMs的能力,导致句子表示次优。在本文中,我们提出一种名为自对比学习(SSCL)的简单方法来缓解此问题,该方法从PLMs的中间层采样负样本,从而提高句子表示的质量。我们提出的方法相当简单,且可以轻松扩展到各种最先进的模型以提升性能,可视为一种即插即用的对比学习框架,用于学习无监督句子表示。大量结果表明,SSCL在语义文本相似性和迁移数据集上为不同强基线(如BERT和SimCSE)带来了显著的性能提升。我们的代码可在https://github.com/nuochenpku/SSCL获取。

英文摘要:

Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task. Experimentally, we observe that the over-smoothing problem reduces the capacity of these powerful PLMs, leading to sub-optimal sentence representations. In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate layers, improving the quality of the sentence representation. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting, which can be seen as a plug-and-play contrastive framework for learning unsupervised sentence representation. Extensive results prove that SSCL brings the superior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets. Our codes are available at https://github.com/nuochenpku/SSCL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑