本文提出了一种新的,高效的迁移学习方法,称为可扩展的权重重新参数化(SWR),是高效和有效的多个下游任务。高效的迁移学习涉及利用在较大数据集上训练的预训练模型,并将其重新用于下游任务,目的是最大限度地重用预训练模型。然而,以前的工作导致了更新参数和特定任务模块的增加,导致了更多的计算,特别是对于微小的模型。此外,没有实际考虑控制更新参数的数量。为了解决这些问题,我们建议学习一个策略网络,它可以决定在哪里重新参数化预训练模型,同时遵守更新参数数量的给定约束。策略网络仅在迁移学习过程中使用,而不是之后。因此,我们的方法在拟议的多语言关键字识别和标准基准(ImageNet to Sketch)中获得了最先进的性能,同时需要零额外计算和显着更少的额外参数。
英文摘要:
This paper proposes a novel, efficient transfer learning method, called Scalable Weight Reparametrization (SWR) that is efficient and effective for multiple downstream tasks. Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks with the aim of maximizing the reuse of the pre-trained model. However, previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. Additionally, there has been no practical consideration for controlling the number of updated parameters. To address these issues, we suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters. The policy network is only used during the transfer learning process and not afterward. As a result, our approach attains state-of-the-art performance in a proposed multi-lingual keyword spotting and a standard benchmark, ImageNet-to-Sketch, while requiring zero additional computations and significantly fewer additional parameters.
Towards Scalable and Versatile Weight Space Learning
University of St.Gallen(圣加仑大学);International Computer Science Institute(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用重点实验室(东南大学),教育部,中国)
ZoRRO: A Zero-Weight Personalized Recommender System for Scalable News Recommendation
Technical University of Denmark(丹麦技术大学);ZOZO Research(ZOZO研究公司);University of California San Diego(加州大学圣地亚哥分校);Pioneer Centre for Artificial Intelligence(先锋人工智能中心)
Self-Supervised Weight Templates for Scalable Vision Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China(计算机科学与工程学院,东南大学,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),教育部,中国)
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);University of British Columbia(不列颠哥伦比亚大学);University of Oxford(牛津大学);University of Edinburgh(爱丁堡大学)
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
The Hong Kong Polytechnic University, PolyU(香港理工大学,PolyU);Hong Kong Polytechnic University Daya Bay Technology(香港理工大学达亚拜技术);Innovation Research Institute(创新研究院)
Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference
KlingAI Research(可灵AI研究院);Fudan University(复旦大学);Shanghai AI Incubation and Innovation Center(上海人工智能孵化与创新中心);Shanghai Academy of AI for Science(上海人工智能科学研究院);University of Science and Technology of China(中国科学技术大学)