发表机构
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education; School of Computer Science and Technology, Beijing Jiaotong University（交通大数据与人工智能重点实验室（北京交通大学），教育部; 北京交通大学计算机科学与技术学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对LoRA共享低秩更新限制token级利用的问题，提出U-LoRA，通过条件门控生成token级利用系数并结合序列上下文约束，辅以偏差校正EMA稳定训练，在不扩展子空间下提升性能。
AI 中文摘要
低秩自适应（LoRA）通过将模型更新限制在低秩子空间内实现参数高效微调，并已在实践中得到广泛应用。然而，LoRA通常对所有token采用共享的低秩更新，这限制了其针对不同序列中的token充分利用自适应子空间的能力。为解决这一问题，我们提出了一种低秩自适应的自适应利用方法（U-LoRA），该方法采用条件门控来显式学习对有限低秩自适应子空间的有效token级利用。具体而言，U-LoRA为每个token生成沿低秩方向的利用系数，并利用序列级上下文信息对其进行联合协调和约束，从而在句子内部诱导出更一致的自适应模式。为进一步增强训练稳定性，我们引入了一种偏差校正的指数移动平均（EMA）历史先验，用于跨优化步骤校准利用信号，抑制由批次间波动引起的噪声。我们方法的有效性源于通过输入条件策略更好地利用现有低秩子空间，而非扩展子空间。在数学推理和自然语言理解基准上的实验表明，在与强LoRA基线及近期变体在可比参数预算下，U-LoRA取得了具有竞争力的性能。
英文摘要
Low-Rank Adaptation (LoRA) achieves parameter-efficient fine-tuning by constraining model updates to a low-rank subspace and has been widely used in practice. However, LoRA typically employs a shared low-rank update across tokens, which limits its ability to fully exploit the adaptation subspace for tokens from different sequences. To address this issue, we propose an adaptive utilization of Low-Rank Adaptation (U-LoRA), which employs conditioned gating to explicitly learn effective token-level utilization of the limited low-rank adaptation subspace. Specifically, U-LoRA generates utilization coefficients along low-rank directions for each token and jointly coordinates and constrains them using sequence-level contextual information, thereby inducing more consistent adaptive patterns within a sentence. To further enhance training stability, we introduce a bias-corrected exponential moving average (EMA) historical prior that calibrates utilization signals across optimization steps, suppressing noise caused by batch-to-batch fluctuations. The effectiveness of our method arises from a better utilization of the existing low-rank subspace via input-conditioned strategies, rather than from expanding the subspace. Experiments on mathematical reasoning and natural language understanding benchmarks demonstrate that U-LoRA achieves competitive performance under comparable parameter budgets when with strong LoRA baselines and recent variants.
CommentsICML 2026
