arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11171cs.CLcs.AI
收藏

语言引导的贝叶斯优化用于高效LoRA超参数搜索

A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

Grad. School of AI, POSTECH, Pohang, Korea(POSTECH人工智能研究生院,韩国坡安);School of EE, KAIST, Daejeon, Korea(韩国科学技术院电子工程学院,韩国大田);School of Computing, KAIST, Daejeon, Korea(韩国科学技术院计算学院,韩国大田)

Baek Seong-Eun, Lee Jung-Mok, Kim Sung-Bin, Tae-Hyun Oh

AI总结:

提出一种利用预训练LLM领域知识的贝叶斯优化框架,通过语言提示将超参数映射到连续空间,结合子集训练代理评估,仅需约30次迭代即可发现比标准超参数提升20%以上性能的LoRA超参数。

摘要:

使用低秩适配(LoRA)微调大型语言模型(LLM)提供了一种资源高效的方式来实现个性化或专业化。然而,LoRA对超参数选择高度敏感,且穷举超参数搜索计算成本高昂。为此,我们提出一个贝叶斯优化(BO)框架,利用预训练LLM的领域知识来高效搜索LoRA超参数。我们的方法将预训练LLM重新用作离散到连续映射模块,将超参数及其领域知识链接到连续向量空间,在其中进行BO。我们通过语言提示设计和控制映射,提供描述超参数间关系及其各自角色的领域感知文本提示。这使我们能够以自然语言将关于LoRA的领域知识显式注入LLM。我们还引入一个额外的可学习标记,以捕获提示中难以用语言描述的残差信息。这有助于BO采样更多高性能超参数。此外,通过利用LoRA训练机制中从完整数据集和子集训练数据集获得的性能之间观察到的强相关性,我们引入使用数据子集的代理训练和评估。这显著提高了我们方法的效率。我们证明,仅需约30次迭代发现的超参数,相比从约45,000种组合中找到的标准超参数,实现了超过20%的性能提升。项目页面:https://baekseongeun.github.io/lora-bo/

英文摘要:

Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) offers a resource-efficient way to personalize or specialize. However, LoRA is highly sensitive to hyperparameter choices, and exhaustive hyperparameter search is computationally expensive. To address this, we propose a Bayesian Optimization (BO) framework that leverages the domain knowledge of pre-trained LLMs to efficiently search for LoRA hyperparameters. Our approach repurposes a pre-trained LLM as a discrete-to-continuous mapping module to link hyperparameters and their domain knowledge to a continuous vector space, where BO is conducted. We design and control the mapping via language prompting, providing a domain-aware textual prompt that describes the relationships among hyperparameters and their respective roles. This allows us to explicitly inject domain knowledge about LoRA into the LLM in natural language. We also introduce an additional learnable token to capture residual information that is difficult to describe linguistically in the prompt. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the strong correlation observed between the performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation using a data subset. This significantly improves the efficiency of our method. We demonstrate that our hyperparameter, discovered with only about 30 iterations, achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations. Project page: https://baekseongeun.github.io/lora-bo/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑