记笔记带来专注？迈向多轮多模态对话学习
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning
AI总结：
本文提出多轮多模态对话数据集MMDiag和模型DiagNote，通过Deliberate与Gaze模块协同进行思维链推理和标注，提升多模态定位与推理能力。
摘要：
多模态大语言模型（MLLMs）建立在大规模预训练的视觉塔和语言模型之上，已在多模态理解方面展现出强大能力。然而，现有大多数MLLMs是在单轮视觉问答任务上训练的，这并不能准确反映真实世界中的人类对话。在本文中，我们引入了MMDiag，一个多轮多模态对话数据集。该数据集通过精心设计的规则和GPT辅助协同生成，具有问题之间、问题与图像之间以及不同图像区域之间的强相关性，因此更贴近真实场景。MMDiag可作为多轮多模态对话学习的强有力基准，并为MLLMs的定位与推理能力带来更多挑战。此外，受人类视觉处理机制的启发，我们提出了DiagNote，一个具备多模态定位与推理能力的MLLM。DiagNote由两个模块（Deliberate和Gaze）组成，二者在多轮对话过程中相互交互，分别执行思维链（Chain-of-Thought）和标注任务。我们通过实验证明了DiagNote在定位以及视觉与语言信息的联合处理与推理方面优于现有MLLMs。
英文摘要：
Multimodal large language models (MLLMs), built on large-scale pre-trained vision towers and language models, have shown great capabilities in multimodal understanding. However, most existing MLLMs are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. In this paper, we introduce MMDiag, a multi-turn multimodal dialogue dataset. This dataset is collaboratively generated through deliberately designed rules and GPT assistance, featuring strong correlations between questions, between questions and images, and among different image regions; thus aligning more closely with real-world scenarios. MMDiag serves as a strong benchmark for multi-turn multimodal dialogue learning and brings more challenges to the grounding and reasoning capabilities of MLLMs. Further, inspired by human vision processing, we present DiagNote, an MLLM equipped with multimodal grounding and reasoning capabilities. DiagNote consists of two modules (Deliberate and Gaze) interacting with each other to perform Chain-of-Thought and annotations respectively, throughout multi-turn dialogues. We empirically demonstrate the advantages of DiagNote in both grounding and jointly processing and reasoning with vision and language information over existing MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
