arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.11297cs.CL
收藏

揭示句子嵌入微调的关键方面:表示秩分析

Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis

LINE Investment Technologies(LINE投资技术公司);Seoul National University(首尔大学);Samsung Advanced Institute of Technology(三星先进技术研究院)

Euna Jung, Jaeill Kim, Jungmin Ko, Jinwoo Park, Wonjong Rhee

AI总结:

本文以表示秩分析句子嵌入的对比学习微调过程,划分两个阶段并揭示关键特性变化,进而提出秩降低策略以提升五种方法的性能与稳定性。

摘要:

句子嵌入无监督学习的最新进展主要涉及在预训练语言模型上采用基于对比学习(CL-based)的微调。在本研究中,我们通过采用表示秩作为主要分析工具,分析最新的句子嵌入方法。我们首先根据表示秩达到峰值的时间定义微调的 Phase 1 和 Phase 2。利用这些阶段,我们进行了全面分析,并在关键方面获得重要发现,包括对齐性与均匀性、语言能力,以及性能与秩之间的相关性。例如,我们发现,随着微调从 Phase 1 过渡到 Phase 2,关键方面的动态变化可能发生显著改变。基于这些发现,我们试验了一种秩降低(RR)策略,该策略有助于对最新的基于 CL 的方法进行快速且稳定的微调。通过实证研究,我们展示了 RR 在提升五种最先进句子嵌入方法的性能和稳定性方面的功效。

英文摘要:

The latest advancements in unsupervised learning of sentence embeddings predominantly involve employing contrastive learning-based (CL-based) fine-tuning over pre-trained language models. In this study, we analyze the latest sentence embedding methods by adopting representation rank as the primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning based on when representation rank peaks. Utilizing these phases, we conduct a thorough analysis and obtain essential findings across key aspects, including alignment and uniformity, linguistic abilities, and correlation between performance and rank. For instance, we find that the dynamics of the key aspects can undergo significant changes as fine-tuning transitions from Phase 1 to Phase 2. Based on these findings, we experiment with a rank reduction (RR) strategy that facilitates rapid and stable fine-tuning of the latest CL-based methods. Through empirical investigations, we showcase the efficacy of RR in enhancing the performance and stability of five state-of-the-art sentence embedding methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑