This paper studies "unsupervised finetuning", the symmetrical problem of the well-known "supervised finetuning". Given a pretrained model and small-scale unlabeled target data, unsupervised finetuning is to adapt the representation pretrained from the source domain to the target domain so that better transfer performance can be obtained. This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain. In this paper, we find the source data is crucial when shifting the finetuning paradigm from supervise to unsupervise, and propose two simple and effective strategies to combine source and target data into unsupervised finetuning: "sparse source data replaying", and "data mixing". The motivation of the former strategy is to add a small portion of source data back to occupy their pretrained representation space and help push the target data to reside in a smaller compact space; and the motivation of the latter strategy is to increase the data density and help learn more compact representation. To demonstrate the effectiveness of our proposed ``unsupervised finetuning'' strategy, we conduct extensive experiments on multiple different target datasets, which show better transfer performance than the naive strategy.
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);University of Göttingen(哥廷根大学);Tsinghua University(清华大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
TRACER: Persistent Regularization for Robust Multimodal Finetuning
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia(墨尔本大学计算机科学与信息系统学院(CIS)、工程与信息技术学院(FEIT))
FACT: A Simple and Efficient Framework for Active Finetuning
Zhejiang (No. 2024C01020)(浙江(No. 2024C01020));National Natural Science Foundation of China (No. 62302031)(中国国家自然科学基金委员会(No. 62302031));Zhejiang Provincial Natural Science Foundation of China (Nos. LQ23F020024 and LZJMZ24D050009)(中国浙江省自然科学基金委员会(Nos. LQ23F020024 and LZJMZ24D050009))
FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing 100190, China, and the University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院计算技术研究所人工智能安全重点实验室,中国科学院(CAS)及中国科学院大学,北京)
Pretrained Reversible Generation as Unsupervised Visual Representation Learning
Xi’an Jiaotong University(西安交通大学);Shanghai AI Laboratory(上海人工智能实验室);SenseTime(商汤科技);The Chinese University of Hong Kong(香港中文大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学)
Unsupervised Keypoints from Pretrained Diffusion Models
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(向量人工智能研究所);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind);Simon Fraser University(西蒙菲莎大学);University of Toronto(多伦多大学)
VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness
Nanjing University(南京大学);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Tsinghua University(清华大学);Panasonic(松下电器(Panasonic));University of California Berkeley(加州大学伯克利分校)
Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model
School of Electronics and Information Engineering, Beihang University(北京航空航天大学电子信息工程学院);University of Liverpool(利物浦大学);Image Processing Center, School of Astronautics, Beihang University(北京航空航天大学宇航学院图像处理中心)