arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.12433cs.CVcs.AIcs.LG
收藏

重新审视基础模型时代的模型拼接

Revisiting Model Stitching In the Foundation Model Era

The Ohio State University(俄亥俄州立大学);Boston University(波士顿大学);Amazon(亚马逊)

Zheda Mai, Ke Zhang, Fu-En Wang, Zixiao Ken Wang, Albert Y. C. Chen, Lu Xia, Min Sun, Wei-Lun Chao, Cheng-Hao Kuo

AI总结:

本文通过系统协议研究视觉基础模型(如CLIP、DINOv2、SigLIP 2)的可拼接性,提出基于目标模型倒数第二层特征匹配损失的拼接方法,并构建VFM拼接树(VST)实现多模态大模型中多个VFM的准确率-延迟权衡。

摘要:

模型拼接通过一个轻量拼接层将一个模型(源)的早期层连接到另一个模型(目标)的后期层,作为表征兼容性的探针。先前工作发现,尽管初始化或目标不同,但基于同一数据集训练的模型仍然是可拼接的(准确率下降可忽略)。我们重新审视在目标、数据和模态组合(例如CLIP、DINOv2、SigLIP 2)上各异的视觉基础模型(VFM)的拼接,并提出问题:异构VFM是否可拼接?我们引入了一个系统协议,涵盖拼接点、拼接层家族、训练损失和下游任务。三个发现浮现:(1)拼接层训练至关重要:传统方法在拼接点匹配中间特征或端到端优化任务损失时难以保持准确率,尤其是在浅层拼接点。(2)通过在目标模型的倒数第二层使用简单的特征匹配损失,异构VFM在视觉任务上变得可靠可拼接。(3)对于深层拼接点,拼接模型可以超越任一组成模型,仅增加少量推理开销(用于拼接层)。基于这些发现,我们进一步提出VFM拼接树(VST),它在多个VFM之间共享早期层同时保留其后期层,为通常利用多个VFM的多模态大语言模型提供了可控的准确率-延迟权衡。综合来看,我们的研究将拼接从诊断探针提升为整合互补VFM优势并定位其表征对齐或分歧点的实用方法。

英文摘要:

Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs stitchable? We introduce a systematic protocol spanning the stitch points, stitch layer families, training losses, and downstream tasks. Three findings emerge. (1) Stitch layer training matters: conventional approaches that match the intermediate features at the stitch point or optimize the task loss end-to-end struggle to retain accuracy, especially at shallow stitch points. (2) With a simple feature-matching loss at the target model's penultimate layer, heterogeneous VFMs become reliably stitchable across vision tasks. (3) For deep stitch points, the stitched model can surpass either constituent model at only a small inference overhead (for the stitch layer). Building on these findings, we further propose the VFM Stitch Tree (VST), which shares early layers across VFMs while retaining their later layers, yielding a controllable accuracy-latency trade-off for multimodal LLMs that often leverage multiple VFMs. Taken together, our study elevates stitching from a diagnostic probe to a practical recipe for integrating complementary VFM strengths and pinpointing where their representations align or diverge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑