arXiv 2412.129322025-03-09cs.CVcs.AI

CoMT：面向大型视觉-语言模型的多模态思维链新基准

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Key Laboratory of Data Intelligence and Advanced Computing in Provincial Universities, Soochow University（苏州大学省级高校数据智能与先进计算重点实验室）；Research Center for SCIR, Harbin Institute of Technology（哈尔滨工业大学SCIR研究中心）；National University of Singapore（新加坡国立大学）

Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, Libo Qin

备注：: Accepted at AAAI 2025; Project Page: https://github.com/czhhzc/CoMT

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CoMT基准，要求LVLM在多模态输入下输出多模态推理结果，涵盖视觉创建、删除、更新和选择四类任务，以揭示现有模型在复杂视觉操作与简洁表达上的能力与局限。

摘要：

大型视觉-语言模型（LVLMs）最近在多模态任务中展现出令人瞩目的成功，包括在多模态思维链（MCoT）推理方面取得的进展。尽管取得了这些成功，当前的基准仍然遵循多模态输入和文本模态输出的传统范式，这导致了显著的缺陷，例如缺少视觉操作和表达含糊。受此启发，我们引入了一个新颖的多模态思维链（CoMT）基准来解决这些局限性。与传统的MCoT基准不同，CoMT同时要求多模态输入和多模态推理输出，旨在模仿类人推理，这种推理天然地整合了视觉操作。具体而言，CoMT包含四个类别：（1）视觉创建，（2）视觉删除，（3）视觉更新，以及（4）视觉选择，以全面探索真实场景中的复杂视觉操作和简洁表达。我们在CoMT上评估了各种LVLM和策略，揭示了当前方法的能力和局限性方面的一些关键见解。我们希望CoMT能够激发更多关于将多模态生成引入推理过程的研究。

英文摘要：

Large Vision-Language Models (LVLMs) have recently demonstrated amazing success in multi-modal tasks, including advancements in Multi-modal Chain-of-Thought (MCoT) reasoning. Despite these successes, current benchmarks still follow a traditional paradigm with multi-modal input and text-modal output, which leads to significant drawbacks such as missing visual operations and vague expressions. Motivated by this, we introduce a novel Chain of Multi-modal Thought (CoMT) benchmark to address these limitations. Different from the traditional MCoT benchmark, CoMT requires both multi-modal input and multi-modal reasoning output, aiming to mimic human-like reasoning that inherently integrates visual operation. Specifically, CoMT consists of four categories: (1) Visual Creation, (2) Visual Deletion, (3) Visual Update, and (4) Visual Selection to comprehensively explore complex visual operations and concise expression in real scenarios. We evaluate various LVLMs and strategies on CoMT, revealing some key insights into the capabilities and limitations of the current approaches. We hope that CoMT can inspire more research on introducing multi-modal generation into the reasoning process.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。