arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.16147cs.CLcs.AI
收藏

TS-PEFT:揭示参数高效微调中的令牌级冗余

TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning

Qfin Holdings, Inc.(Qfin控股公司)

Dabiao Ma, Ziming Dai, Zhimin Xin, Shu Wang, Jian Yang, Haojun Fei

AI总结:

TS-PEFT通过动态探针揭示令牌级冗余,优化令牌更新以提升微调效率,优于传统基线方法。

摘要:

当前参数高效微调(PEFT)方法通常基于一个隐含假设:一旦选定目标模块,通过该模块的每个令牌对下游任务的贡献是均等的,都需要参数更新。在本文中,我们挑战这一传统,揭示了在大型模型(LMs)微调过程中普遍存在令牌级冗余。我们提出了TS-PEFT,一个利用近端优化的理论框架,它在微调过程中充当动态探针,以识别令牌级冗余。广泛实验表明,无差别更新所有令牌不仅计算上是多余的,而且往往引入优化噪声。令人惊讶的是,通过丢弃30%-70%的令牌更新,TS-PEFT始终能够匹配或超过密集基线如LoRA、DoRA的性能。我们的深入分析显示,所学习的令牌级稀疏性比传统权重标准是更优越的模块重要性指标,提供了一种新的数据驱动视角,以理解LMs的内在适应机制。

英文摘要:

Current Parameter-Efficient Fine-Tuning (PEFT) methods typically operate under an implicit assumption: Once a target module is selected, every token passing through it contributes equally to the downstream task and requires a parameter update. In this paper, we challenge this convention by revealing a pervasive token-level redundancy in the fine-tuning of large models (LMs). We propose TS-PEFT, a theoretical framework utilizing proximal optimization that acts as a dynamic probe to identify token-level redundancy during the fine-tuning process. Extensive experiments demonstrate that indiscriminately updating all tokens is not only computationally superfluous but often introduces optimization noise. Surprisingly, by discarding 30%-70% of token updates, TS-PEFT consistently matches or exceeds the performance of dense baselines such as LoRA, DoRA. Our in-depth analysis shows that the learned token-level sparsity is a superior indicator of module importance compared to traditional weight criteria, providing a novel data-driven perspective on the intrinsic adaptation mechanism of LMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑