M$^3$IT：面向多模态多语言指令微调的大规模数据集
M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
AI总结：
针对开放视觉语言模型高质量多语言多模态指令数据集稀缺的问题，研究者推出含240万样本、覆盖80种语言的M³IT数据集，并训练Ying-VLM验证其效果，数据集已开源。
摘要：
指令微调已显著推动了ChatGPT等大语言模型（LLM）的发展，使其能够在各类任务中与人类指令对齐。然而，由于高质量指令数据集稀缺，开放视觉语言模型（VLM）的发展受到了限制。为应对这一挑战并推动视觉语言领域的研究，我们推出了多模态多语言指令微调（M$^3$IT）数据集，旨在优化视觉语言模型与人类指令的对齐效果。我们的M$^3$IT数据集包含40个精心整理的子数据集，涵盖240万个样本和400条人工撰写的任务指令，均被重构为视觉到文本的格式。核心任务通过先进的翻译系统被译为80种语言，确保了更广泛的可用性。M$^3$IT在任务覆盖度、指令数量和样本规模上均超越了现有数据集。此外，我们开发了Ying-VLM——一个基于M$^3$IT数据集训练的视觉语言模型，验证了其在回答需世界知识的复杂问题、泛化到未见过的视频任务以及理解未见过的中文指令方面的潜力。我们已开源该数据集以推动后续研究。
英文摘要：
Instruction tuning has significantly advanced large language models (LLMs) such as ChatGPT, enabling them to align with human instructions across diverse tasks. However, progress in open vision-language models (VLMs) has been limited due to the scarcity of high-quality instruction datasets. To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M$^3$IT) dataset, designed to optimize VLM alignment with human instructions. Our M$^3$IT dataset comprises 40 carefully curated datasets, including 2.4 million instances and 400 manually written task instructions, reformatted into a vision-to-text structure. Key tasks are translated into 80 languages with an advanced translation system, ensuring broader accessibility. M$^3$IT surpasses previous datasets regarding task coverage, instruction number and instance scale. Moreover, we develop Ying-VLM, a VLM model trained on our M$^3$IT dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese. We have open-sourced the dataset to encourage further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
