重新思考微调：解锁视觉-语言模型中的隐藏能力
Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
AI总结：
本文提出基于MFT的结构重参数化方法，通过重新组织VLMs内部子网络实现高效微调，超越LoRA和全微调，无需改变骨干网络。
摘要：
在对视觉-语言模型（VLMs）进行微调的探索中，如来自参数高效微调（PEFT）的低秩适应（LoRA）等方法已取得了显著进展。然而，大多数方法依赖于显式的权重更新，忽视了在预训练模型中已编码的广泛表征结构，这些结构仍然未被充分利用。最近的研究表明，掩码微调（MFT）可以成为语言模型的一种强大且高效的后训练范式。与更新权重不同，MFT为每个权重分配可学习的门控分数，使模型能够重新组织其内部子网络以适应下游任务。在本文中，我们从基于MFT的结构重参数化角度重新思考VLMs的微调。我们对具有不同语言骨干网络的VLMs的语言和投影组件应用MFT，并与强大的PEFT基线进行比较。实验表明，MFT在各方面都超过了LoRA变体，甚至超过了全微调，实现了高性能而无需改变冻结的骨干网络。我们的发现表明，有效的适应不仅可以通过更新权重实现，还可以通过重新建立模型现有知识之间的连接来实现。代码可在：https://github.com/Ming-K9/MFT-VLM 上获得。
英文摘要：
Fine-tuning has become the dominant paradigm for adapting Vision-Language Models (VLMs), yet most approaches rely on explicit weight updates that introduce a fundamental trade-off. Full Fine-Tuning (FFT) may perturb pretrained representations due to cross-modal gradient interference, whereas Parameter-Efficient Fine-Tuning (PEFT) methods rely on additive modules, such as low-rank adapters, which may limit adaptation capacity. In this paper, we rethink VLM adaptation from a structural selection framework that adapts VLMs without modifying backbone weights, and we propose Mask Fine-Tuning (MFT). MFT learns masks that selectively route information through existing pretrained connections, dynamically uncovering subnetworks that better align pretrained representations with downstream objectives. Extensive experiments show that MFT provides an effective structural alternative to both FFT and PEFT, consistently achieving superior performance across multiple vision-language benchmarks without adding knowledge or altering the deployment architecture. Moreover, our analysis with MFT provides new insights into how pretrained VLMs reorganize their internal representational pathways during adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
