PALO: 面向50亿人口的多语言大型多模态模型
PALO: A Polyglot Large Multimodal Model for 5B People
AI总结：
本研究推出支持10种语言、覆盖50亿人口的多语言多模态模型PALO，通过半自动化翻译构建多语言指令数据集，训练三种规模模型均超基线，还发布首个相关评测基准。
摘要：
为构建更具包容性的视觉-语言模型（VLM），本研究推出名为PALO的大型多语言多模态模型。PALO支持10种主要语言的视觉推理能力，涵盖英语、中文、印地语、西班牙语、法语、阿拉伯语、孟加拉语、俄语、乌尔都语和日语，覆盖总计约50亿人口（占世界人口的65%）。我们的方法采用半自动化翻译策略，借助经微调的大语言模型将多模态指令数据集从英语适配至目标语言，既确保高语言保真度，又因人工工作量极低而具备可扩展性。引入多样化指令集有助于提升我们在多种语言上的整体性能，尤其是印地语、阿拉伯语、孟加拉语和乌尔都语这类代表性不足的语言。最终训练出三种参数规模（17亿、70亿和130亿参数）的模型，以验证其泛化性与可扩展性，我们观察到其相较强基线模型有显著提升。我们还提出了首个多语言多模态基准，供后续研究评估其跨语言的视觉-语言推理能力。代码地址：https://github.com/mbzuai-oryx/PALO。
英文摘要：
In pursuit of more inclusive Vision-Language Models (VLMs), this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
