A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
Grad. School of AI, POSTECH, Pohang, Korea(POSTECH人工智能研究生院,韩国坡安);School of EE, KAIST, Daejeon, Korea(韩国科学技术院电子工程学院,韩国大田);School of Computing, KAIST, Daejeon, Korea(韩国科学技术院计算学院,韩国大田)
Baek Seong-Eun, Lee Jung-Mok, Kim Sung-Bin, Tae-Hyun Oh
Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) offers a resource-efficient way to personalize or specialize. However, LoRA is highly sensitive to hyperparameter choices, and exhaustive hyperparameter search is computationally expensive. To address this, we propose a Bayesian Optimization (BO) framework that leverages the domain knowledge of pre-trained LLMs to efficiently search for LoRA hyperparameters. Our approach repurposes a pre-trained LLM as a discrete-to-continuous mapping module to link hyperparameters and their domain knowledge to a continuous vector space, where BO is conducted. We design and control the mapping via language prompting, providing a domain-aware textual prompt that describes the relationships among hyperparameters and their respective roles. This allows us to explicitly inject domain knowledge about LoRA into the LLM in natural language. We also introduce an additional learnable token to capture residual information that is difficult to describe linguistically in the prompt. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the strong correlation observed between the performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation using a data subset. This significantly improves the efficiency of our method. We demonstrate that our hyperparameter, discovered with only about 30 iterations, achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations. Project page: https://baekseongeun.github.io/lora-bo/
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong(香港大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of California, Los Angeles(加州大学洛杉矶分校);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Cornell University(康奈尔大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Oak Ridge National Laboratory(橡树岭国家实验室);Yale University(耶鲁大学);University of California, Irvine(加州大学欧文分校);Tulane University(路易斯安那大学塔兰帕伊斯分校)
Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
School of Electronics and Computer Science, University of Southampton, UK(索姆塞特大学电子与计算机科学学院);Department of Computer Science, University of Liverpool, UK(利弗pool大学计算机科学系)
Flexora: Flexible Low Rank Adaptation for Large Language Models
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));School of Information Technology, Carleton University(卡尔顿大学信息科技学院)
First Token Probability Guided RAG for Telecom Question Answering
Shenzhen Research Institute of Big Data(深圳大数据研究院);School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)理工学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
School of Computing and Information Technology, Great Bay University(计算与信息学院,大亚湾大学);Beijing Institute of Technology(北京理工大学);University of Warwick(沃里克大学);City University of Hong Kong (Dongguan)(香港城市大学(东莞));Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));College of Computer Science and Software Engineering, Shenzhen University(计算机科学与软件工程学院,深圳大学)
Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation
National Engineering Research Center For Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);School of Computer Science, Peking University(北京大学计算机学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算科学与数据科学学院);Centre for Medical Informatics, University of Edinburgh(爱丁堡大学医学信息中心);Health Data Research UK(英国健康数据研究)