零样本视觉编码器嫁接：基于LLM代理模型
Zero-Shot Vision Encoder Grafting via LLM Surrogates
AI总结：
提出零样本视觉编码器嫁接方法，通过小型代理模型训练编码器后直接迁移至大型LLM，降低训练成本约45%，且性能与完整训练相当。
摘要：
视觉语言模型（VLM）通常将规模适中的视觉编码器与大型语言模型（LLM）配对，例如Llama-70B，这使得解码器成为训练过程中的主要计算负担。为降低成本，一种有前景的策略是首先使用小型语言模型训练视觉编码器，然后将其迁移至大型语言模型。我们构建了小型“代理模型”，通过直接继承大型目标LLM的浅层，使其与大型目标LLM共享相同的嵌入空间和表示语言。在代理模型上训练的视觉编码器可以直接迁移至更大的模型，我们将这一过程称为零样本嫁接——当直接接入完整规模的目标LLM时，嫁接后的配对性能超过编码器-代理模型配对，并且在某些基准上甚至与使用目标LLM进行完整解码器训练的性能相当。此外，当使用Llama-70B作为解码器时，我们的代理训练方法将整体VLM训练成本降低了约45%。代码位于https://github.com/facebookresearch/zero。
英文摘要：
Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden during training. To reduce costs, a potential promising strategy is to first train the vision encoder using a small language model before transferring it to the large one. We construct small "surrogate models" that share the same embedding space and representation language as the large target LLM by directly inheriting its shallow layers. Vision encoders trained on the surrogate can then be directly transferred to the larger model, a process we call zero-shot grafting -- when plugged directly into the full-size target LLM, the grafted pair surpasses the encoder-surrogate pair and, on some benchmarks, even performs on par with full decoder training with the target LLM. Furthermore, our surrogate training approach reduces overall VLM training costs by ~45% when using Llama-70B as the decoder. The code is at https://github.com/facebookresearch/zero.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
