Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models. In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side. We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet. Diff-Tuning achieves a 26% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning.
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
The University of Hong Kong(香港大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Hong Kong Generative AI Research and Development Center(香港生成式人工智能研究与开发中心)
On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning
Department of Electrical and Computer Engineering, The University of British Columbia(英国不列颠哥伦比亚大学电气与计算机工程系);Department of Mathematics, The University of British Columbia(英国不列颠哥伦比亚大学数学系);Department of Chemical and Biological Engineering, The University of British Columbia(英国不列颠哥伦比亚大学化学与生物工程系)
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Huawei Technologies Co.,Ltd, China(华为技术有限公司);Shenzhen International Center for Industrial and Applied Mathematics(深圳国际工业与应用数学中心)
Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
Department of Computer Science, Iowa State University(爱荷华州立大学计算机科学系);Department of Computer Science, Kalinga Institute of Industrial Technology(卡林加工业技术学院计算机科学系);Department of Civil, Construction & Environmental Engineering, Iowa State University(爱荷华州立大学土木、建筑与环境工程系)
Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
Department of Electrical and Computer Engineering, Isfahan University of Technology(伊斯法罕理工大学电气与计算机工程系);RIKEN Center for Advanced Intelligence Project (AIP)(理化学研究所先进智能项目中心);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所);School of Computer Science, Simon Fraser University(西蒙菲莎大学计算机科学学院);Sydney AI Centre, The University of Sydney(悉尼大学悉尼人工智能中心);Department of Computer Science, Hong Kong Baptist University(香港浸会大学计算机科学系)
Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
IDEAS NCBR;University of Warsaw(华沙大学);Warsaw University of Technology(华沙理工大学);Jagiellonian University(雅盖隆大学);Google DeepMind(谷歌DeepMind);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所)
Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies
Jožef Stefan Institute(约热夫· Stefan 研究所);Jožef Stefan International Postgraduate School(约热夫· Stefan 国际研究生学院);Faculty of Computer and Information Science, University of Ljubljana(卢布尔雅那大学计算机与信息科学学院)
本研究比较大语言模型在跨语言零样本与全样本设置下的 adapter/全参数微调及 IT、CLV 迁移策略,发现 IT 更利于目标语言,CLV 更能保留英语知识。
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
Effective Unbounded Drift Control in Heterogeneous Federated Learning: Bayesian Fine-Tuning Using a Reference Model
School of Computer Science and Engineering(计算机科学与工程学院);Center for Biosystems and Biotech Data Science(生物系统与生物技术数据科学中心);Ghent University Global Campus, Incheon, Republic of Korea(因斯布鲁克大学全球校区,韩国)
Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州市先进研究 institute,中国科学技术大学);School of Computer Science and Technology, China University of Mining and Technology(计算机科学与技术学院,中国矿业大学);Mine Digitization Engineering Research Center of the Ministry of Education(教育部矿山数字化工程研究中心)
Mila - Quebec AI Institute(魁北克人工智能研究所);Polytechnique Montréal(蒙特利尔理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);CIFAR AI Chair(CIFAR人工智能主席);Google DeepMind(谷歌DeepMind)
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Division of Statistics and Machine Learning (STIMA), Department of Computer and Information Science (IDA), Linköping University(统计与机器学习系(STIMA)、计算机与信息科学系(IDA)、利厄普堡大学)