本文综述 model reprogramming(模型重编程),通过复用源领域预训练模型且免微调解决目标领域任务,以应对资源受限场景下的数据、成本和预训练模型不足问题。
摘要:
在视觉、语言和语音等数据丰富的领域,深度学习普遍用于交付高性能的特定任务模型,甚至能够学习通用的、与任务无关的表示,以便高效微调到下游任务。然而,资源受限领域中的深度学习仍面临多重挑战,包括:(i)数据有限,(ii)模型开发成本受限,以及(iii)缺乏用于有效微调的充足预训练模型。本文概述了 model reprogramming(模型重编程),以弥合这一差距。模型重编程通过重新调整用途并复用来自源领域的成熟预训练模型来解决目标领域中的任务,且无需模型微调,从而实现资源高效的跨领域机器学习;其中源领域与目标领域可能差异巨大。在许多应用中,模型重编程的表现优于迁移学习和从零开始训练。本文阐明了模型重编程的方法论,总结了现有用例,对模型重编程成功的原因给出了理论解释,并在最后讨论了开放式研究问题与机遇。有关模型重编程研究的列表正在 https://github.com/IBM/model-reprogramming 上持续维护和更新。
英文摘要:
In data-rich domains such as vision, language, and speech, deep learning prevails to deliver high-performance task-specific models and can even learn general task-agnostic representations for efficient finetuning to downstream tasks. However, deep learning in resource-limited domains still faces multiple challenges including (i) limited data, (ii) constrained model development cost, and (iii) lack of adequate pre-trained models for effective finetuning. This paper provides an overview of model reprogramming to bridge this gap. Model reprogramming enables resource-efficient cross-domain machine learning by repurposing and reusing a well-developed pre-trained model from a source domain to solve tasks in a target domain without model finetuning, where the source and target domains can be vastly different. In many applications, model reprogramming outperforms transfer learning and training from scratch. This paper elucidates the methodology of model reprogramming, summarizes existing use cases, provides a theoretical explanation of the success of model reprogramming, and concludes with a discussion on open-ended research questions and opportunities. A list of model reprogramming studies is actively maintained and updated at https://github.com/IBM/model-reprogramming.
Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition
Georgia Institute of Technology(佐治亚理工学院);Research Center for Information Technology Innovation, Academia Sinica(中央研究院资讯科技创新研究中心);Kore University of Enna(恩纳科雷大学);IBM Research(IBM研究院)
Reprogramming Foundational Large Language Models(LLMs) for Enterprise Adoption for Spatio-Temporal Forecasting Applications: Unveiling a New Era in Copilot-Guided Cross-Modal Time Series Representation Learning
BasisN: Reprogramming-Free RRAM-Based In-Memory-Computing by Basis Combination for Deep Neural Networks
Technical University of Munich(慕尼黑工业大学);Technical University of Darmstadt(达姆施塔特工业大学);Zhejiang University(浙江大学);National Cheng Kung University(成功大学);University of Siegen(锡根大学)
Explainable Machine Learning Reveals 12-Fold Ucp1 Upregulation and Thermogenic Reprogramming in Female Mouse White Adipose Tissue After 37 Days of Microgravity: First AI/ML Analysis of NASA OSD-970
Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);NTT Communication Science Laboratories(NTT通信科学实验室);RIKEN Center for Advanced Intelligence Project (AIP)(理化学研究所革新智能综合研究中心)
Cross-lingual transfer of multilingual models on low resource African Languages
School of Computer Science and Engineering, Vellore Institute of Technology(韦洛尔理工学院计算机科学与工程学院);Department of Computer Science, University of Pretoria(比勒陀利亚大学计算机科学系)