Alleviating Over-smoothing for Unsupervised Sentence Representation
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);Microsoft(微软);Duke University(杜克大学);Peking University(北京大学)
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Bowen Cao, Jianhui Chang, Daxin Jiang, Jia Li
Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task. Experimentally, we observe that the over-smoothing problem reduces the capacity of these powerful PLMs, leading to sub-optimal sentence representations. In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate layers, improving the quality of the sentence representation. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting, which can be seen as a plug-and-play contrastive framework for learning unsupervised sentence representation. Extensive results prove that SSCL brings the superior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets. Our codes are available at https://github.com/nuochenpku/SSCL.
Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Kuaishou Technology(快手科技)
Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correction
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));University of Illinois Chicago(芝加哥大学伊利诺伊分校)
Smoothed Contrastive Learning for Unsupervised Sentence Embedding
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Kuaishou Technology(快手科技)
Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)