arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.13093cs.CL
收藏

负样本对对比学习的影响--以句子嵌入为例

Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding

Northwest University(西北大学);Shaanxi Normal University(陕西师范大学);University of Leeds(利兹大学)

Rui Cao, Yihao Wang, Yuxin Liang, Ling Gao, Jie Zheng, Jie Ren, Zheng Wang

摘要:

对比学习是一种从未标记数据中提取知识的强大技术。该技术需要两种成分的平衡混合物:阳性(相似)和阴性(不相似)样本。这通常通过在训练期间维护阴性样本的队列来实现。该领域的先前工作通常使用固定长度的负样本队列,但负样本大小如何影响模型性能仍不清楚。在采用对比学习时,负样本数量对性能的不透明影响引起了我们的深入探讨。提出了一种用于句子嵌入的负样本队列动量对比学习模型MoCoSE。通过在在线分支上增加预测层,使模型具有非对称性,并结合目标分支的EMA更新机制,防止模型崩溃。定义了一个最大可追踪距离度量,通过该度量可以了解文本对比学习在多大程度上受益于负样本的历史信息。实验结果表明,当最大可追踪距离在一定范围内时,查询结果最好,证明了对于负样本队列,历史信息存在一个最优范围。我们评估建议的无监督MoCoSE的语义文本相似性(STS)的任务,并获得平均斯皮尔曼的相关性$77.27\%$。源代码可在https://github.com/xbdxwyh/mocose上获得。

英文摘要:

Contrastive learning is emerging as a powerful technique for extracting knowledge from unlabeled data. This technique requires a balanced mixture of two ingredients: positive (similar) and negative (dissimilar) samples. This is typically achieved by maintaining a queue of negative samples during training. Prior works in the area typically uses a fixed-length negative sample queue, but how the negative sample size affects the model performance remains unclear. The opaque impact of the number of negative samples on performance when employing contrastive learning aroused our in-depth exploration. This paper presents a momentum contrastive learning model with negative sample queue for sentence embedding, namely MoCoSE. We add the prediction layer to the online branch to make the model asymmetric and together with EMA update mechanism of the target branch to prevent the model from collapsing. We define a maximum traceable distance metric, through which we learn to what extent the text contrastive learning benefits from the historical information of negative samples. Our experiments find that the best results are obtained when the maximum traceable distance is at a certain range, demonstrating that there is an optimal range of historical information for a negative sample queue. We evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS) task and obtain an average Spearman's correlation of $77.27\%$. Source code is available at https://github.com/xbdxwyh/mocose.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑