发表机构
Dalian University of Technology; Northeastern University; Hebei Key Laboratory of Marine Perception Network and Data Processing; Nanjing Tech University; Institute of Intelligent Manufacturing（大连理工大学; 东北大学; 河北省海洋感知网络与数据处理重点实验室; 南京工业大学; 智能制造研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对LoRA传统秩分配问题，提出RSLoRA，通过激活空间几何及虚拟表示探测机制确定高敏感性模块，无需训练调整和反向梯度，在主流基准测试中优于现有方法，为模型自适应提供高效方案。
AI 中文摘要
低秩自适应（LoRA）已成为参数高效微调（PEFT）的基石，然而传统的均匀秩分配忽略了神经层的功能异质性。现有秩分配方法在计算强度和启发式简单性之间难以权衡。本文提出RSLoRA，一种由激活空间几何驱动的无训练、无梯度的秩分配器。通过识别层间的“敏感性状态转移”，引入虚拟表示探测机制，利用有效秩和弗雷歇距离确定高敏感性模块。实验表明RSLoRA在主流基准测试中优于现有方法，为大规模模型自适应提供了高效、稳健且感知表示的解决方案。
英文摘要
Parameter-efficient fine-tuning enables large language models to adapt to downstream tasks with substantially lower computational and storage cost, and Low-Rank Adaptation (LoRA) is among its most widely used techniques. However, vanilla LoRA assigns a uniform rank to all adapted modules, while existing adaptive methods either incur additional optimization overhead or rely on static weights and local gradients that do not capture task-conditioned representation changes. We propose RSRA, a training-free rank allocator that estimates where adaptation capacity is most needed through forward-only representation sensitivity probing on a small calibration set. Specifically, RSRA uses Spectral Effective Rank to allocate capacity across layers, measures module-wise hidden-state displacement under standardized virtual low-rank updates with the Frechet Distance, and combines both signals through hierarchical normalization to produce a task-aware rank configuration before fine-tuning. Across commonsense reasoning and natural language understanding benchmarks with Qwen3-4B and Mistral-7B, RSRA achieves the highest average performance in all three reported model-benchmark settings and a 1.48x-1.93x speedup in allocation time over the fastest competing pre-allocation method. When integrated with DoRA, LoRA-FA, and PiSSA, RSRA improves 15 of the 18 evaluated combinations and increases the average performance of all three PEFT methods.
Comments14 pages, 5 figures
