多模态自指导：利用语言模型合成抽象图像与视觉推理指令
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
AI总结：
本文提出多模态自指导方法，利用大语言模型合成抽象图像与视觉推理指令，构建涵盖八类场景的基准并微调LMM，揭示现有模型不足并提升图表理解与地图导航能力。
摘要：
尽管当前大多数大型多模态模型（LMMs）已经能够理解自然场景照片和肖像，但它们对抽象图像（例如图表、地图或布局）的理解以及视觉推理能力仍然相当初步。它们常常在简单的日常任务中遇到困难，例如从时钟读取时间、理解流程图或使用道路地图规划路线。鉴于此，我们设计了一种多模态自指导方法，利用大型语言模型及其代码能力，在日常生活场景中大规模合成抽象图像和视觉推理指令。我们的策略轻松构建了一个包含11,193条指令的多模态基准，覆盖八种视觉场景：图表、表格、模拟地图、仪表盘、流程图、关系图、平面图和视觉谜题。该基准仅使用简单线条和几何元素构建，揭示了Claude-3.5-Sonnet和GPT-4o等最先进LMMs在抽象图像理解、空间关系推理和视觉元素归纳方面的不足。此外，为了验证合成数据的质量，我们使用62,476条合成的图表、表格和道路地图指令对一个LMM进行了微调。结果表明，该模型在图表理解和地图导航性能上有所提升，同时也展示了在其他视觉推理任务上的潜在益处。我们的代码可在以下地址获取：https://github.com/zwq2018/Multi-modal-Self-instruct。
英文摘要：
Although most current large multimodal models (LMMs) can already understand photos of natural scenes and portraits, their understanding of abstract images, e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite rudimentary. They often struggle with simple daily tasks, such as reading time from a clock, understanding a flowchart, or planning a route using a road map. In light of this, we design a multi-modal self-instruct, utilizing large language models and their code capabilities to synthesize massive abstract images and visual reasoning instructions across daily scenarios. Our strategy effortlessly creates a multimodal benchmark with 11,193 instructions for eight visual scenarios: charts, tables, simulated maps, dashboards, flowcharts, relation graphs, floor plans, and visual puzzles. \textbf{This benchmark, constructed with simple lines and geometric elements, exposes the shortcomings of most advanced LMMs} like Claude-3.5-Sonnet and GPT-4o in abstract image understanding, spatial relations reasoning, and visual element induction. Besides, to verify the quality of our synthetic data, we fine-tune an LMM using 62,476 synthetic chart, table and road map instructions. The results demonstrate improved chart understanding and map navigation performance, and also demonstrate potential benefits for other visual reasoning tasks. Our code is available at: \url{https://github.com/zwq2018/Multi-modal-Self-instruct}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
