Parrot：多语言视觉指令微调
Parrot: Multilingual Visual Instruction Tuning
AI总结：
针对多模态大语言模型在多语言处理上的退化问题，提出PARROT方法，利用文本引导和专家混合进行视觉标记对齐，并构建多语言基准MMMB，取得最先进性能。
摘要：
多模态大语言模型（MLLMs）的快速发展，如GPT-4o，标志着向人工通用智能迈出的重要一步。现有方法通常通过监督微调（SFT）将视觉编码器与LLMs对齐，但这往往随着训练进程而损害其处理多种语言的能力。我们通过实验观察到，不平衡的SFT数据集（主要以英语为中心）由于多语言标记对齐的失败，会降低非英语语言的性能。为解决这一问题，我们提出了PARROT，一种新颖的方法，利用文本引导在语言层面进行视觉标记对齐。PARROT将视觉标记条件化于多样化的语言输入，并使用专家混合（MoE）来对齐多语言标记。通过计算初始视觉特征与文本嵌入之间的交叉注意力，我们选择最相关的专家，将视觉标记转换为特定语言的表示。此外，我们引入了大规模多语言多模态基准（MMMB），这是一个包含6种语言、15个类别和12,000个问题的新基准，用于评估多语言能力。PARROT在多语言基准和广泛的多模态任务上均达到了最先进的性能。代码和数据集可在 https://github.com/AIDC-AI/Parrot 获取。
英文摘要：
The rapid development of Multimodal Large Language Models (MLLMs), such as GPT-4o, marks a significant step toward artificial general intelligence. Existing methods typically align vision encoders with LLMs via supervised fine-tuning (SFT), but this often deteriorates their ability to handle multiple languages as training progresses. We empirically observe that imbalanced SFT datasets, largely English-centric, degrade performance on non-English languages due to the failure in multilingual token alignment. To address this, we propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level. PARROT conditions visual tokens on diverse language inputs and uses Mixture-of-Experts (MoE) to align multilingual tokens. By computing cross-attention between initial visual features and textual embeddings, we select the most relevant experts, converting visual tokens into language-specific representations. Additionally, we introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions, to assess multilingual capabilities. PARROT achieves state-of-the-art performance on both the multilingual benchmarks and a wide range of multimodal tasks. Code and dataset are available at: https://github.com/AIDC-AI/Parrot
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
