发表机构
University of California, Riverside; Pinterest（加州大学河滨分校; Pinterest）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出SilK方法，利用聚类验证指标作为自监督目标，无需负样本或图数据，实现高效文本表示微调，在速度和内存上显著优于基线，且下游任务性能保持竞争力。
AI 中文摘要
自监督微调在无标签的情况下优化预训练语言编码器的嵌入空间。然而，常用的方法计算成本高昂。具体而言，基于对比学习的方法需要多视图数据和批内负样本，而无负样本的方法则需要辅助图或网络。一个有趣的问题随之产生：能否在不依赖额外负样本或图数据的情况下进行自监督微调？为回答此问题，我们提出了SilK（基于轮廓的K均值），它训练于聚类验证指标，即一种无需标签的聚类质量内部度量。SilK对语料库进行聚类，然后将简化轮廓回归至目标值。每个文档仅与k个聚类中心比较，绝不与其他文档比较，因此该方法无需数据增强、无需负样本对，且每个文档只需一个视图。在BERT-base上，SilK每轮训练速度比我们评估的最快基线快1.46倍，峰值GPU内存比最精简基线少45.4%。在冻结编码器线性探测下，SilK在三个下游任务上与最佳基线保持竞争力。
英文摘要
Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.
