模态间与模态内度量：视觉-语言模型微调结果的预测视角
The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Models
AI总结：
提出模态间与模态内度量（IIMM），通过量化模态内相似性与模态间错位的关系，预测视觉-语言模型微调后的性能变化，并提供理论界限，实现轻量级可迁移性预测。
摘要：
大型视觉-语言基础模型的微调仍是一个未被充分探索的领域，特别是关于其对学习增益和灾难性遗忘的影响。受对比双编码器中模态间隙重要性的启发，我们引入了模态间与模态内度量（IIMM）——一种预测性指标，用于量化模态内图像嵌入相似性与模态间错位之间的关系。通过对四种最先进的视觉-语言模型和五种微调技术进行广泛的实证分析，我们建立了一个强线性关系：IIMM分数较高的任务在领域内性能提升更大，但会遭受更显著的领域外性能下降，其中一些参数高效微调（PEFT）方法表现出严重的遗忘。与现有的可迁移性度量相比，IIMM在双编码器模型中微调后准确性变化方面展现出显著更强的预测能力。此外，我们提供了一个理论界限，证明IIMM的变化受限于微调前后嵌入分布之间的Wasserstein距离，从而确保其作为预测度量的稳定性和鲁棒性。仅需对目标数据进行一次前向传播，从业者即可利用这一关键见解来评估模型在微调后期望提升的程度。当与模型在多样化任务上的先前性能知识相结合时，IIMM进一步增强了新任务的可迁移性预测，为引导模型适应策略提供了一种轻量级且有效的工具。我们的代码可在https://github.com/mit-ll/IIMM获取。
英文摘要：
The fine-tuning of large vision-language foundation models remains an underexplored area, particularly regarding its impact on learning gains and catastrophic forgetting. Inspired by the significance of modality gaps in contrastive dual-encoders, we introduce the Inter-Intra Modal Measure (IIMM) - a predictive metric that quantifies the relationship between intra-modal image embedding similarity and inter-modal misalignment. Through extensive empirical analysis across four state-of-the-art vision-language models and five fine-tuning techniques, we establish a strong linear relationship: tasks with higher IIMM scores yield greater in-domain performance improvements but suffer from more pronounced out-of-domain degradation, with some parameter-efficient fine-tuning (PEFT) methods exhibiting severe forgetting. Compared to existing transferability measures, the IIMM demonstrates significantly stronger predictive power for accuracy changes post fine-tuning in dual-encoder models. Moreover, we provide a theoretical bound, proving that changes in IIMM are limited by the Wasserstein distance between pre- and post-fine-tuning embedding distributions, ensuring its stability and robustness as a predictive measure. With only a single forward pass of the target data, practitioners can leverage this key insight to evaluate the degree to which a model can be expected to improve following fine-tuning. When combined with prior knowledge of a model's performance across diverse tasks, the IIMM further enhances transferability predictions for novel tasks, offering a lightweight yet effective tool for guiding model adaptation strategies. Our code is provided at https://github.com/mit-ll/IIMM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
