LLaMA-Adapter V2：参数高效的视觉指令模型
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
AI总结：
针对LLaMA-Adapter泛化开放式视觉指令能力不足的问题，提出LLaMA-Adapter V2，通过解锁更多可学习参数、早期视觉融合、联合训练等策略，以14M新增参数实现更强的多模态及纯语言指令遵循能力。
摘要：
如何高效将大语言模型（LLM）转化为指令遵循模型是近期热门研究方向，而针对多模态推理训练LLM的探索仍相对不足。尽管近期的LLaMA-Adapter展现了用LLM处理视觉输入的潜力，但它仍无法很好地泛化到开放式视觉指令任务，且性能落后于GPT-4。本文提出LLaMA-Adapter V2，一种参数高效的视觉指令模型。具体而言，我们首先通过解锁更多可学习参数（如归一化层、偏置项和缩放因子）对LLaMA-Adapter进行增强，将指令遵循能力分散到除适配器外的整个LLaMA模型中。其次，我们提出早期融合策略，仅将视觉标记输入到LLM的早期层，助力更好地整合视觉知识。第三，通过优化不相交的可学习参数组，引入图文对与指令遵循数据的联合训练范式。该策略有效缓解了图文对齐和指令遵循两项任务间的干扰，仅需小规模图文与指令数据集即可实现强大的多模态推理能力。推理阶段，我们将额外的专家模型（如图像描述/OCR系统）整合到LLaMA-Adapter中，在不产生训练成本的情况下进一步提升其图像理解能力。与原始LLaMA-Adapter相比，我们的LLaMA-Adapter V2仅需在LLaMA基础上新增14M参数，即可执行开放式多模态指令。这一新设计的框架还展现出更强的纯语言指令遵循能力，甚至在聊天交互中表现出色。我们的代码和模型可在https://github.com/ZrrSkywalker/LLaMA-Adapter 获取。
英文摘要：
How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
