arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09658cs.LGcs.AIcs.CV
收藏

任务向量跨预训练模型传输的梯度符号遮蔽

Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models

University of Modena and Reggio Emilia(摩德纳和雷吉奥艾米利亚大学);Vector Institute(向量研究所);University of Toronto(多伦多大学)

Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Fengyuan Liu, Marco Ciccone, Angelo Porrello, Simone Calderara

AI总结:

GradFix通过近似梯度符号结构,利用少量标记样本实现任务向量跨预训练模型的高效转移,提升多任务和多源模型性能。

摘要:

当一个新的基础模型发布时,从业者通常需要重复微调,即使相同任务已在之前的版本中被解决。一个有前途的替代方法是重用捕捉模型如何适应特定任务的参数变化(即任务向量)。然而,这些向量往往无法在不同的预训练模型之间转移,因为它们的参数空间存在错位。在本工作中,我们表明成功的转移强烈依赖于新模型的梯度符号结构。基于这一见解,我们提出了GradFix,它近似了理想的符号结构,并利用它仅使用少量标记样本进行知识转移。值得注意的是,这不需要额外的微调:我们只计算几个目标模型的梯度而无需参数更新,并相应地遮蔽源任务向量。这产生了一个与目标损失景观局部对齐的更新,有效地将任务向量重新定位到新的预训练上。我们提供了一个理论保证,我们的方法确保了一阶下降。经验上,我们在视觉和语言基准上展示了显著的性能提升,持续优于朴素的任务向量添加和少样本微调。我们进一步显示,传输任务向量可以改善多任务和多源模型融合。代码可在https://github.com/fillo-rinaldi/GradFix上获得。

英文摘要:

When a new release of a foundation model is published, practitioners typically need to repeat fine-tuning, even if the same task was already tackled in the previous version. A promising alternative is to reuse the parameter changes (i.e., task vectors) that capture how a model adapts to a specific task. However, these vectors often fail to transfer across different pre-trained models because their parameter spaces are misaligned. In this work, we show that successful transfer depends strongly on the gradient-sign structure of the new model. Based on this insight, we propose GradFix, which approximates the ideal sign structure and leverages it to transfer knowledge using only a handful of labeled samples. Notably, this requires no additional fine-tuning: we only compute a few target-model gradients without parameter updates and mask the source task vector accordingly. This yields an update that is locally aligned with the target loss landscape, effectively rebasing the task vector onto the new pre-training. We provide a theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant performance gains on vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning. We further show that transporting task vectors improves multi-task and multi-source model merging. Code is available at https://github.com/fillo-rinaldi/GradFix.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑