Genesis:用于多模态思维链推理的视觉增强原语
Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning
AI总结:
本研究针对多模态大模型视觉推理在复杂真实场景下性能下降的问题,提出视觉增强原语VAP及相关方法,构建UNREAL基准,验证了视觉推理的缩放定律。
摘要:
近期多模态大语言模型在视觉推理任务上取得了令人鼓舞的结果,在现有STEM基准上接近人类水平的性能。然而,这些能力在复杂的真实场景下会下降。我们发现,思维链推理在很大程度上仍局限于语言空间,很少将每一步锚定在视觉证据上,主要原因是训练过程中缺乏此类数据。在这项工作中,我们引入了一种训练数据增强流水线——视觉增强原语(Visual Augmentation Primitives, VAP),它利用最先进的视觉生成模型合成多样化的逼真训练样本及对应的推理文本。VAP引入了一些基础算子,将其应用于种子图像得到变体,还在整个生成过程中构建连贯的多模态推理链。这类训练数据迫使模型感知视觉内容,并在每一步根据视觉内容进行逻辑推理。我们进一步引入基于多模态思维链准则的强化学习来增强该方法,它不仅奖励文本推理,还通过对中间推理步骤的逻辑有效性、完整性和一致性进行评分来奖励视觉推理。此外,我们构建了UNREAL基准,以系统评估不同可控难度级别的超真实场景下的视觉推理。实验表明,基于该框架构建的4B模型超过了8B基线,且随着模型规模扩展至32B,性能持续提升,验证了视觉推理的缩放定律。
英文摘要:
Recent multimodal large language models achieve promising results on visual reasoning tasks, approaching human-level performance on existing STEM benchmarks. However, these capabilities degrade under complex real-world scenes. We identify that chain-of-thought reasoning remains largely confined to the linguistic space, rarely anchoring each step in visual evidence. The main cause is the absence of such data during training. In this work, we introduce a training data augmentation pipeline, \textbf{Visual Augmentation Primitives} (VAP), which leverages state-of-the-art vision generative models to synthesize diverse photorealistic training samples and corresponding reasoning text. VAP introduces some basic operators and applies them to a seed image and gets a variant. It also constructs coherent multimodal reasoning chains throughout the generation process. Such training data force models to perceive visual content and perform logical reasoning according to them at every step. We further introduce multimodal chain-of-thought rubric-based reinforcement learning to enhance this. It rewards not only text reasoning but also visual reasoning by scoring intermediate reasoning steps for logical validity, completeness, and consistency. Moreover, we construct the UNREAL benchmark to systematically evaluate visual reasoning on hyperrealistic scenes across controllable difficulty levels. Experiments show that a 4B model built on this framework surpasses 8B baselines, with performance improving consistently as the model scales to 32B, validating the scaling law for visual reasoning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。