Fine-tuning from a collection of models pre-trained on different domains (a "model zoo") is emerging as a technique to improve test accuracy in the low-data regime. However, model selection, i.e. how to pre-select the right model to fine-tune from a model zoo without performing any training, remains an open topic. We use a linearized framework to approximate fine-tuning, and introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation. Since all model selection algorithms in the literature have been tested on different use-cases and never compared directly, we introduce a new comprehensive benchmark for model selection comprising of: i) A model zoo of single and multi-domain models, and ii) Many target tasks. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models. We show our model selection baseline can select optimal models to fine-tune in few selections and has the highest ranking correlation to fine-tuning accuracy compared to existing algorithms.
Fast Linearized Alternating Direction Minimization Algorithm with Adaptive Parameter Selection for Multiplicative Noise Removal
Department of Mathematics, Third Military Medical University, Chongqing 400038, Chongqing, People's Republic of China（数学系，第三军医大学）；Department of Mathematics and System, School of Sciences, National University of Defense Technology, Changsha 410073, Hunan, People's Republic of China（数学与系统系，国防科技大学）
Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
School of Statistics and Data Science, KLMDASR, LEBPS, and LPMC, Nankai University（统计与数据科学学院，KLMDASR，LEBPS和LPMC，南开大学）；Image Processing Center, School of Astronautics, Beihang University（图像处理中心，航空学院，北航）；School of Computer Science and Technology, Tiangong University（计算机科学与技术学院，天工大学）
TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning
Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Renmin University of China（中国人民大学）；Suzhou Key Laboratory of Artificial Intelligence（苏州人工智能重点实验室）
Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models
West China Hospital, Sichuan University（四川大学华西医院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；College of Computer Science, Sichuan University（四川大学计算机学院）；School of Biomedical Engineering, University of Science and Technology of China（中国科学技术大学生物医学工程学院）
CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Hangzhou Dianzi University（杭州电子科技大学）；School of Computer Science (National Pilot Software Engineering School), BUPT（计算机学院（国家级试点软件工程学院），北京邮电大学）
ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning
East China Normal University（华东师范大学）；Zhongguancun Academy（中关村学院）；The Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
