自主想象：多模态大模型视觉推理中视觉-文本转换的闭环分解
Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Models
AI总结：
针对多模态大模型视觉-文本转换存在感知瓶颈、难完成简单视觉任务的问题，提出自主想象方法，通过闭环迭代修改视觉输入生成中间状态来分解转换过程，无需重训即可让模型解决超出原有感知能力的任务。
摘要：
在纯文本模态下，大语言模型（LLMs）已能通过将复杂推理任务分解为更简单的子问题，在这类任务中取得显著成效。然而，多模态大语言模型（MLLMs）在一些看似简单的视觉任务上仍存在困难，比如计数和拼图求解。我们认为这些任务对视觉-文本转换能力构成挑战——在该过程中，MLLMs将从输入场景感知到的视觉信息转换为文本信息，以用于后续推理和生成答案。如果视觉输入的复杂度超出了MLLMs的感知能力，不分解这一转换过程的话，仅扩展推理时的推理量无法解决任务，因为模型会反复遭遇相同的感知瓶颈。我们提出了“自主想象（autonomous imagination）”方法，使MLLMs能够迭代修改视觉输入（例如隔离物体、重新排列拼图块）以生成中间视觉状态，将视觉-文本转换分解为闭环的视觉修改步骤。实验表明，无需任何重新训练，MLLMs现在就能解决最初超出其感知能力的任务，这说明闭环视觉修改是将视觉推理任务分解为可解决子步骤的有效方式。我们的代码和数据已发布在https://future-item.github.io/autoimagine-site/。
英文摘要：
Under pure textual modality, Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seemingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of visual-to-textual conversion, where MLLMs convert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck. We propose an approach, autonomous imagination, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at https://future-item.github.io/autoimagine-site/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
