AbductiveMLLM: 提升多模态大语言模型中的视觉归纳推理
AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs
AI总结:
AbductiveMLLM通过结合REASONER和IMAGINER组件,提升多模态大语言模型在视觉归纳推理中的性能。
摘要:
视觉归纳推理(VAR)是一项具有挑战性的任务,要求人工智能系统为不完整的视觉观察推断最可能的解释。尽管最近的多模态大语言模型(MLLMs)在通用多模态推理能力方面表现出色,但在归纳推理方面仍无法与人类相比。为了弥合这一差距,我们受到人类认知中言语与图像归纳之间相互作用的启发,提出通过模仿这种双模式行为来增强MLLMs的归纳能力。具体而言,我们引入了AbductiveMLLM,包含两个协同工作的组件:REASONER和IMAGINER。REASONER在言语领域运作。它首先使用盲目的LLM探索可能的解释的广阔空间,然后根据跨模态因果对齐剔除视觉不一致的假设。剩下的假设被引入MLLM作为定向先验,引导其推理朝向因果一致的解释。另一方面,IMAGINER通过模仿人类般的图像思维进一步引导MLLMs。它将文本到图像扩散模型条件化在输入视频和REASONER的输出嵌入上,以“想象”出与言语解释相对应的合理视觉场景,从而丰富MLLMs的上下文基础。这两个组件以端到端的方式联合训练。在标准VAR基准测试中,AbductiveMLLM实现了最先进的性能,一致地优于传统解决方案和先进的MLLMs。
英文摘要:
Visual abductive reasoning (VAR) is a challenging task that requires AI systems to infer the most likely explanation for incomplete visual observations. While recent MLLMs develop strong general-purpose multimodal reasoning capabilities, they fall short in abductive inference, as compared to human beings. To bridge this gap, we draw inspiration from the interplay between verbal and pictorial abduction in human cognition, and propose to strengthen abduction of MLLMs by mimicking such dual-mode behavior. Concretely, we introduce AbductiveMLLM comprising of two synergistic components: REASONER and IMAGINER. The REASONER operates in the verbal domain. It first explores a broad space of possible explanations using a blind LLM and then prunes visually incongruent hypotheses based on cross-modal causal alignment. The remaining hypotheses are introduced into the MLLM as targeted priors, steering its reasoning toward causally coherent explanations. The IMAGINER, on the other hand, further guides MLLMs by emulating human-like pictorial thinking. It conditions a text-to-image diffusion model on both the input video and the REASONER's output embeddings to "imagine" plausible visual scenes that correspond to verbal explanation, thereby enriching MLLMs' contextual grounding. The two components are trained jointly in an end-to-end manner. Experiments on standard VAR benchmarks show that AbductiveMLLM achieves state-of-the-art performance, consistently outperforming traditional solutions and advanced MLLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。