arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.09510cs.CVcs.LG
收藏

无监督微调

Unsupervised Finetuning

University of Science and Technology of China(中国科学技术大学);Microsoft Cloud & AI(微软云与人工智能)

Suichan Li, Dongdong Chen, Yinpeng Chen, Lu Yuan, Lei Zhang, Qi Chu, Bin Liu, Nenghai Yu

摘要:

本文研究了著名的“有监督的微调”的对称性问题“无监督微调”。给定一个预训练的模型和小规模未标记的目标数据,无监督微调是将从源域预训练的表示调整到目标域,以便获得更好的传输性能。这个问题比有监督的问题更具挑战性,因为小规模目标数据的低数据密度对于无监督学习来说是不友好的,导致预训练表示的损坏和目标域的差表示。在本文中,我们发现,源数据是关键的微调范式从监督到无监督,并提出了两个简单而有效的策略,将源数据和目标数据相结合的无监督微调:“稀疏源数据重放”和“数据混合”。前一种策略的动机是将一小部分源数据添加回以占据其预训练的表示空间,并帮助推动目标数据驻留在更小的紧凑空间中;后一种策略的动机是增加数据密度,并帮助学习更紧凑的表示。为了证明我们提出的“无监督微调”策略的有效性,我们对多个不同的目标数据集进行了广泛的实验,这些数据集显示出比朴素策略更好的传输性能。

英文摘要:

This paper studies "unsupervised finetuning", the symmetrical problem of the well-known "supervised finetuning". Given a pretrained model and small-scale unlabeled target data, unsupervised finetuning is to adapt the representation pretrained from the source domain to the target domain so that better transfer performance can be obtained. This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain. In this paper, we find the source data is crucial when shifting the finetuning paradigm from supervise to unsupervise, and propose two simple and effective strategies to combine source and target data into unsupervised finetuning: "sparse source data replaying", and "data mixing". The motivation of the former strategy is to add a small portion of source data back to occupy their pretrained representation space and help push the target data to reside in a smaller compact space; and the motivation of the latter strategy is to increase the data density and help learn more compact representation. To demonstrate the effectiveness of our proposed ``unsupervised finetuning'' strategy, we conduct extensive experiments on multiple different target datasets, which show better transfer performance than the naive strategy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑