arXiv 2605.195232026-05-19cs.CLcs.AIcs.CV

探究跨模态技能注入：场景、方法与超参数

Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters

State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）

Zhiyu Xu, Lean Wang, Yuanxin Liu, Lei Li, Hao Zhou, Fandong Meng, Jie Zhou, Xu Sun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究了跨模态技能注入在不同场景下的表现，分析了其方法和超参数的影响，发现其在指令遵循和跨语言任务中表现良好，但在数学推理中存在困难，同时指出经典方法如TA和DARE在性能上优于其他融合方法。

摘要：

视觉-语言模型（VLMs）在一般多模态理解方面表现出色；然而，它们在高效获取持续演化的领域特定技能方面存在困难。传统增强VLM能力的方法，如监督微调（SFT），需要大量的数据集整理和大量的计算资源。模型合并作为一种高效的替代方法，能够将领域专家的LLM专业知识转移到VLMs上，而无需额外的数据集要求或显著的计算开销。与传统合并同质LLM的方法不同，跨模态技能注入旨在通过将领域专家LLM整合到VLM中来诱导出新的跨模态能力。然而，现有研究缺乏对跨模态技能注入的适用性和方法的系统分析。在本研究中，我们从三个主要方面探讨了跨模态技能注入：场景、方法和超参数。在场景方面，我们发现跨模态技能注入在指令遵循和跨语言设置中表现良好，但在数学推理中表现不佳。在方法方面，我们发现经典方法如TA和DARE在性能上优于其他融合方法。我们还提供了这些经典方法所依赖的超参数调优的系统和定量分析。

英文摘要：

Vision-Language Models (VLMs) have demonstrated remarkable proficiency in general multi-modal understanding; yet they struggle to efficiently acquire continually evolving domain-specific skills. Conventional approaches to enhancing VLM capabilities, such as Supervised Fine-Tuning (SFT), require extensive dataset curation and substantial computational resources. Model merging has emerged as an efficient alternative that enables the transfer of domain-specific expertise from Large Language Models (LLMs) to VLMs without incurring additional training data requirements or significant computational overhead. Unlike conventional merging of homogeneous LLMs, which mainly aggregates existing capabilities, cross-modal skill injection aims to induce emergent cross-modal capabilities by integrating a domain-expert LLM into a VLM. However, existing research lacks a systematic analysis of the applicability and methodology of cross-modal skill injection. In this study, we investigate cross-modal skill injection across three main aspects: scenarios, methods, and hyperparameters. For scenarios, we find that cross-modal skill injection generally performs well in instruction-following and cross-lingual settings, yet struggles with mathematical reasoning. For methods, we find that classic approaches such as TA and DARE consistently achieve superior performance over alternative merging methods. We also provide a systematic and quantitative analysis of the hyperparameter tuning that these classic methods critically depend on.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。