arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.04682cs.CLcs.AI
收藏

TiTok: 通过对比超额迁移LoRA

TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA

Yonsei University(延世大学)

Chanjoo Jung, Jaehyung Kim

AI总结:

TiTok通过token级知识迁移实现有效LoRA迁移,利用源模型有无LoRA的token对比超额提取任务相关信息,无需额外模型即可选择性过滤合成数据,在多个迁移设置中实现平均+4~10%的性能提升。

摘要:

大型语言模型(LLMs)在现实场景中广泛应用,但微调需高昂的计算和存储成本。参数高效微调(PEFT)方法如LoRA降低了这些成本;然而,适应的参数依赖于基础模型,无法跨不同backbone迁移。知识蒸馏是一种解决方案,但其效果依赖于训练数据。近期的TransLoRA通过生成合成数据避免了这一问题,但需要额外训练判别器模型。本文提出TiTok,一种通过token级知识迁移实现有效LoRA迁移的新框架。具体而言,TiTok通过源模型有无LoRA的token级对比超额提取任务相关信息,突出信息性token并选择性过滤合成数据,无需额外模型或开销。通过在三个基准上的多个迁移设置实验,证明TiTok在整体上平均比基线提升+4~10%。

英文摘要:

Large Language Models (LLMs) are widely applied in real world scenarios, yet fine-tuning them comes with significant computational and storage costs. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA mitigate these costs; however, the adapted parameters are dependent on the base model and cannot be transferred across different backbones. One way to address this issue is through knowledge distillation, but its effectiveness inherently depends on training data. Recent work such as TransLoRA avoids this by generating synthetic data; nevertheless, this adds complexity since it requires training an additional discriminator model. In this paper, we propose TiTok, a new framework that enables effective LoRA Transplantation through Token-level knowledge transfer. Specifically, TiTok captures task-relevant information through a token-wise contrastive excess between a source model with and without LoRA. This excess highlights informative tokens and enables selective filtering of synthetic data, all without additional models or overhead. Through experiments on three benchmarks across multiple transfer settings, we demonstrate that TiTok is consistently effective, achieving average performance gains of +4~10% compared to baselines overall.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑