Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；Southern University of Science and Technology（南方科技大学）；University of Warwick（华威大学）
Tao Li, Weisen Jiang, Fanghui Liu, Xiaolin Huang, James T. Kwok
Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by "model soups"~\cite{wortsman2022model} via exploring various hyperparameter configurations.The Learned-Soup, a variant of model soups, significantly improves the performance but suffers from substantial memory and time costs due to the requirements of (i) having to load all fine-tuned models simultaneously, and (ii) a large computational graph encompassing all fine-tuned models. In this paper, we propose Memory Efficient Hyperplane Learned Soup (MEHL-Soup) to tackle this issue by formulating the learned soup as a hyperplane optimization problem and introducing block coordinate gradient descent to learn the mixing coefficients. At each iteration, MEHL-Soup only needs to load a few fine-tuned models and build a computational graph with one combined model. We further extend MEHL-Soup to MEHL-Soup+ in a layer-wise manner. Experimental results on various ViT models and data sets show that MEHL-Soup(+) outperforms Learned-Soup(+) in terms of test accuracy, and also reduces memory usage by more than $13\times$. Moreover, MEHL-Soup(+) can be run on a single GPU and achieves $9\times$ speed up in soup construction compared with the Learned-Soup. The code is released at https://github.com/nblt/MEHL-Soup.
TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs
Faculty of Engineering, The University of Sydney（悉尼大学工程学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络与空间安全学院）
Tracking Tensor Subspaces with Informative Random Sampling for Real-Time MR Imaging
Stanford University（斯坦福大学）；University of Minnesota（明尼苏达大学）；Center for Magnetic Resonance Research（磁共振研究中心）；University of Minnesota Medical School（明尼苏达大学医学院）
Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs
Tianjin University National Center for Applied Mathematics（天津大学应用数学中心）；Jilin University School of Mechanical and Aerospace Engineering（吉林大学机械与 aerospace 工程学院）
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Mila & DIRO, Université de Montréal（蒙特利尔大学米拉与DIRO）；Critical ML, University of Waterloo（滑铁卢大学Critical ML）；Zhejiang University（浙江大学）；Canada CIFAR AI Chair（加拿大CIFAR AI主席）
TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement
