Multitask learning is a methodology to boost generalization performance and also reduce computational intensity and memory usage. However, learning multiple tasks simultaneously can be more difficult than learning a single task because it can cause imbalance among tasks. To address the imbalance problem, we propose an algorithm to balance between tasks at the gradient level by applying gradient-based meta-learning to multitask learning. The proposed method trains shared layers and task-specific layers separately so that the two layers with different roles in a multitask network can be fitted to their own purposes. In particular, the shared layer that contains informative knowledge shared among tasks is trained by employing single gradient step update and inner/outer loop training to mitigate the imbalance problem at the gradient level. We apply the proposed method to various multitask computer vision problems and achieve state-of-the-art performance.
KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education(东南大学计算机网络和信息集成教育部重点实验室);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Shenzhen University of Advanced Technology(深圳大学);Pengcheng Laboratory(鹏城实验室);Sun Yat-sen University(中山大学);Southern University of Science and Technology(南方科技大学);George Washington University(乔治华盛顿大学);University of Chinese Academy of Sciences(中国科学院大学)
Dual-Balancing for Physics-Informed Neural Networks
Department of Computer Science, Hong Kong Baptist University(香港 Baptist 大学计算机科学系);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(IHPC),科技研究局(A*STAR))