How do we transfer the relevant knowledge from ever larger foundation models into small, task-specific downstream models that can run at much lower costs? Standard transfer learning using pre-trained weights as the initialization transfers limited information and commits us to often massive pre-trained architectures. This procedure also precludes combining multiple pre-trained models that learn complementary information. To address these shortcomings, we introduce Adaptive Feature Transfer (AFT). Instead of transferring weights, AFT operates purely on features, thereby decoupling the choice of the pre-trained model from the smaller downstream model. Rather than indiscriminately compressing all pre-trained features, AFT adaptively transfers pre-trained features that are most useful for performing the downstream task, using a simple regularization that adds minimal overhead. Across multiple vision, language, and multi-modal datasets, AFT achieves significantly better downstream performance compared to alternatives with a similar computational cost. Furthermore, AFT reliably translates improvement in pre-trained models into improvement in downstream performance, even if the downstream model is over $50\times$ smaller, and can effectively transfer complementary information learned by multiple pre-trained models.
Southeast University(东南大学);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education(新一代人工智能技术及其跨学科应用重点实验室(东南大学),教育部)
Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Models
Sun Yat-sen University(中山大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);National Supercomputing Center in Shenzhen(深圳国家超算中心);Huawei Technologies Co., Ltd(华为技术有限公司);Tsinghua University(清华大学)
Parameter Importance-Driven Continual Learning for Foundation Models
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(未来区块链与隐私计算先进创新中心,北京航空航天大学);School of Artificial Intelligence, Beihang University(人工智能学院,北京航空航天大学)
Harvesting AlphaEarth: Benchmarking the Geospatial Foundation Model for Agricultural Downstream Tasks
Department of Earth System Science and Center on Food Security and the Environment, Stanford University, USA(地球系统科学系和食品安全与环境中心,斯坦福大学,美国);Corteva Agriscience™, USA(科特维亚农业科学公司,美国)
ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy
Ningbo Global Innovation Center, Zhejiang University(浙江大学宁波全球创新中心);Tsinghua University(清华大学);Youtu, Tencent(腾讯优图);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院)
DEFT: Disentanglement-Enhanced Fine-Tuning for EEG Foundation Models
Uploading Inc(Uploading公司);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Public Health, Peking University(北京大学公共卫生学院);Microsoft Research Asia(微软亚洲研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
Graduate School of Medicine, Kyoto University(京都大学医学研究科);Medical Data Vision Co., Ltd.(医疗数据视觉株式会社);IQVIA Solutions Japan G.K.(IQVIA日本解决方案公司);DeSC Healthcare, Inc.(DeSC医疗公司)
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
Department of Physics, University of Toronto and Vector Institute(物理系,多伦多大学和向量研究所);NERSC, Lawrence Berkeley National Laboratory(NERSC,伯克利国家实验室);Institut für Experimentalphysik, Universität Hamburg(实验物理研究所,汉堡大学);Nagoya University, Kobayashi-Maskawa Institute(名古屋大学,小林昭夫研究所);Department of Particle Physics and Astrophysics, Stanford University(粒子物理与天体物理系,斯坦福大学);Fundamental Physics Directorate, SLAC National Accelerator Laboratory(基础物理局,SLAC国家加速器实验室)
Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models
KTH Royal Institute of Technology(瑞典皇家理工学院);University of Amsterdam(阿姆斯特丹大学);Digital Futures(数字未来机构);Helmholtz AI(亥姆霍兹人工智能研究所);Technical University of Munich(慕尼黑工业大学);MPI for Intelligent Systems, Tübingen(蒂宾根马克斯·普朗克智能系统研究所)