Model Balancing Helps Low-data Training and Fine-tuning
Dartmouth College(达特茅斯学院);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣地亚哥分校);Lawrence Berkeley National Lab(劳伦斯伯克利国家实验室)
Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang
备注:
EMNLP 2024 Oral. First two authors contributed equally
Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models underscore the importance of low-data training and fine-tuning. This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML). To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers. To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks. Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases. Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an "add-on" method for improving model performance.
Data-driven configuration tuning of glmnet for balancing accuracy and computational efficiency
Joint Graduate School of Mathematics for Innovation, Kyushu University(九州大学数学创新联合研究生院);Institute of Mathematics for Industry, Kyushu University(九州大学产业数学研究所)
SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Xiongan AI Institute(雄安人工智能研究院)
Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心)
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China(计算机科学与技术学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China, China(苏州先进研究院,中国科学技术大学)
Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
Warwick Medical School, University of Warwick(华威大学华威医学院);WMG, University of Warwick(华威大学华威制造集团);Department of Statistics, University of Warwick(华威大学统计系);Alan Turing Institute(艾伦·图灵研究所)
Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Nanjing University(南京大学);Airon Technology CO., LTD(艾瑞森技术有限公司);University of Bristol(布里斯托大学);The University of Oklahoma(俄克拉荷马大学);Donghua University(东华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Carnegie Mellon University(卡内基梅隆大学)
Enhancing Metabolic Syndrome Prediction with Hybrid Data Balancing and Counterfactuals
College of Health Solutions, Arizona State University(亚利桑那州立大学健康解决方案学院);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院)