基于对比学习的句子表示学习中的各向同性、语境化与学习动态研究
On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning
AI总结：
本文从各向同性、语境化和学习动态角度分析对比学习在句子表示学习中的作用机制，揭示其促进表示各向同性、提高句内相似性并缓解虚假语境化，为未来方法设计提供依据。
摘要：
在句子表示学习（SRL）中引入对比学习目标，已在许多句子级自然语言处理任务上带来了显著提升。然而，对比学习为何能够有效学习句子级语义，目前仍缺乏清晰的理解。本文旨在通过从各向同性、语境化和学习动态的视角深入审视基于对比学习的句子表示学习，为未来句子表示学习方法的设计提供指导。我们从表示偏移的几何结构角度解释其成功之处，并表明对比学习带来了各向同性，同时推动了较高的句内相似性：当处于同一句子中时，词元会在语义空间中收敛到相近的位置。我们还发现，我们所形式化的“虚假语境化”在语义上有意义的词元上得到缓解，而在功能性词元上则被增强。我们发现，在训练过程中，嵌入空间被导向原点，更多区域因此得到更好的界定。我们通过观察不同训练温度、批大小和池化方法下的学习动态，对这些发现进行了消融验证。
英文摘要：
Incorporating contrastive learning objectives in sentence representation learning (SRL) has yielded significant improvements on many sentence-level NLP tasks. However, it is not well understood why contrastive learning works for learning sentence-level semantics. In this paper, we aim to help guide future designs of sentence representation learning methods by taking a closer look at contrastive SRL through the lens of isotropy, contextualization and learning dynamics. We interpret its successes through the geometry of the representation shifts and show that contrastive learning brings isotropy, and drives high intra-sentence similarity: when in the same sentence, tokens converge to similar positions in the semantic space. We also find that what we formalize as "spurious contextualization" is mitigated for semantically meaningful tokens, while augmented for functional ones. We find that the embedding space is directed towards the origin during training, with more areas now better defined. We ablate these findings by observing the learning dynamics with different training temperatures, batch sizes and pooling methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
