发表机构
NC State University; University of North Carolina at Chapel Hill（北卡罗来纳州立大学; 北卡罗来纳大学教堂山分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究探讨PortLLM在持续预训练中LoRA补丁的长期时间可移植性及有效性。通过对Mistral、Gemma和Qwen基础模型进行实证研究，并提供理论分析，发现其可移植性持久，高维向量近正交性是关键，还展示了损失景观几何视角。
AI 中文摘要
微调已广泛用于使大语言模型适应特定领域任务。参数高效微调（PEFT）方法如低秩适应（LoRA）常被用于降低计算成本。PortLLM是一种在持续预训练后用于适应大语言模型的无需训练和数据的方案。虽然初始结果显示LoRA补丁有短期时间可移植性，但PortLLM在多次持续预训练更新中的长期性能仍未充分探索，其有效性也缺乏理论理解。我们通过对PortLLM补丁在10个持续预训练步骤中的长期时间可移植性进行广泛实证研究，并提供两种理论分析来解决这两个问题。实证发现可移植性在更长时间持续存在，理论发现高维向量的近正交性是时间可移植性的关键依据，分析还展示了损失景观的几何视角以促进不同适应选项的理论比较。
英文摘要
Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computational costs. PortLLM is a training-free and data-free scheme used to adapt LLMs after continual pretraining. Although the initial PortLLM results show that LoRA patches exhibit short-term temporal portability, the long-term performance of PortLLM across several updates of continual pretraining remains underexplored. Furthermore, the intriguing effectiveness of PortLLM is not well understood from a theoretical standpoint. We address these two open questions by (1) performing an extensive empirical study of the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen; and (2) offering two theoretical analyses to explain our observation that the simple PortLLM method achieves competitive performance. We find empirically that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated. We find theoretically that near-orthogonality of high-dimensional vectors is a key justification for temporal portability. Our analyses also demonstrate a geometric perspective of the loss landscape in facilitating the theoretical comparison of different adaptation options.
