arXiv 2306.054252023-06-08cs.CVcs.AIcs.CLcs.HC

MIMIC-IT：多模态情境下指令微调

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

Nanyang Technological University（南洋理工大学）；Microsoft Research（微软研究院）

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu

备注：: Project page: https://otter-ntu.github.io/ Dataset & code: https://github.com/Luodian/otter Initial release, work in progress

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言指令数据不足的问题，提出含280万对数据的MIMIC-IT数据集及Syphus自动标注流程，训练的Otter模型在多模态任务上表现优异并已开源。

摘要：

高质量的指令与响应对大语言模型在交互式自然语言任务上的零样本性能至关重要。对于涉及复杂视觉场景的交互式视觉语言任务，大量多样且富有创造性的指令-响应对对于调优视觉语言模型（VLM）应是必不可少的。然而，目前可用的视觉语言指令-响应对在数量、多样性和创造性方面仍然有限，给交互式VLM的泛化带来了挑战。\n在此我们提出多模态情境下指令微调（MultI-Modal In-Context Instruction Tuning，MIMIC-IT），这是一个包含280万条多模态指令-响应对的数据集，其中有220万条源自图像和视频的独特指令。每一对都附带多模态情境信息，形成对话上下文，旨在增强VLM的感知、推理和规划能力。被称为Syphus的指令-响应收集流程，通过结合人类专业知识与GPT能力的自动标注流水线实现规模化扩展。\n利用MIMIC-IT数据集，我们训练了一个名为Otter的大型VLM。基于在多个视觉语言基准上开展的广泛评估，观察到Otter在多模态感知、推理和情境学习方面展现出卓越的能力。人类评估显示它能有效契合用户意图。我们发布了MIMIC-IT数据集、指令-响应收集流水线、基准测试集以及Otter模型。

英文摘要：

High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。