Rank-Constrained Adaptation for Reliable Real-World Performance
Department of Electrical Engineering \& Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA;Harvard School of Medicine, Harvard, Boston, MA, USA
Deep learning models trained to optimize average accuracy often exhibit systematic failures on particular subpopulations. In real-world settings like healthcare, the subpopulations most affected by such disparities are frequently unlabeled, partially observed, or not known in advance. Existing group-robust methods typically assume prior knowledge of the relevant subgroups, using group annotations for training, validation, or model selection. We propose Misclassification Aware Rank-Limited Adaptation (MARLA), a parameter-efficient method for improving worst group performance without explicit subgroup annotations. MARLA leverages an ERM-trained model by calculating the model's misclassification probability scores on a held-out adaptation set to identify a low-dimensional subspace where errors concentrate. We then learn a rank-restricted additive correction to the classifier logits within that subspace. Across seven real-world datasets, we evaluate group robustness under three settings: no knowledge of subgroup relevance, partial knowledge of subgroup relevance, and full knowledge of subgroup relevance. MARLA improves worst-group performance while remaining fast and parameter-efficient, with data-guided hyperparameter selection.
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
ML Research, Morgan Stanley, New York, NY 10019, USA(ML研究,摩根大通,纽约,纽约州,10019,美国);Visa Research, Austin, TX 78759, USA(Visa研究,奥斯汀,德克萨斯州,78759,美国);Department of ECE, University of Minnesota, MN 55455, USA(电子工程系,明尼苏达大学,明尼苏达州,55455,美国)
Chien-Shiung Wu College, Southeast University(陈希ung Wu学院,东南大学);MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University(人工智能MOE实验室,人工智能研究院,上海交通大学计算机科学学院);School of Computer Science and Engineering, Southeast University(计算机科学与工程学院,东南大学);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用关键实验室(东南大学),教育部,中国)
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(越南国家大学胡志明市分校科学大学);Vietnam National University, Ho Chi Minh City(越南国家大学胡志明市分校);Trivita AI
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation
Tsinghua University(清华大学);Bauhaus-Universität Weimar(魏玛包豪斯大学);Leibniz University Hannover(汉诺威莱布尼茨大学);CNPC Engineering Technology RD Company Limited(中国石油集团工程技术研究院有限公司);Queensland University of Technology(昆士兰科技大学)
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation
NSF AI Institute for Artificial Intelligence and Fundamental Interactions(NSF人工智能与基础交互AI研究所);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学)
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology(华中科技大学);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学);Shenzhen University(深圳大学)
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Institute of Technology(北京理工大学);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室)
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心);Helmholtz Munich(海德堡-穆恩医疗中心);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);MDSI(慕尼黑数据科学研究所)