弥合多模态基础模型与世界模型之间的差距
Bridging the Gap Between Multimodal Foundation Models and World Models
AI总结：
针对当前多模态基础模型难以胜任世界模型的问题，该研究通过增强结构化推理能力、提出图像视频可控生成框架并拓展至4D生成，推动二者差距的弥合。
摘要：
人类通过整合多种感官模态来理解世界，从而能够感知、推理并想象动态物理过程。受这一能力启发，多模态基础模型（MFMs）已成为多模态理解与生成领域的强大工具。然而，当前的MFMs尚无法充当有效的世界模型，它们缺乏反事实推理、动力学模拟、时空信息理解、生成视觉结果控制以及多维度推理等核心能力。本文探究了弥合多模态基础模型与世界模型差距所需的条件。首先，我们通过判别式任务提升MFMs的推理能力，为其配备因果推断、反事实思维、时空推理等结构化推理技能，使其超越表面相关性，理解视觉与文本数据中的深层关联。其次，我们探索MFMs在图像和视频模态下的生成能力，提出结构化可控生成的新框架。我们的方法结合场景图、多模态条件控制与多模态对齐策略来引导生成过程，确保生成结果符合高层语义与细粒度用户意图。我们进一步将这些技术拓展至可控4D生成，实现跨时空的交互式、可编辑、可变形物体合成。
英文摘要：
Humans understand the world through the integration of multiple sensory modalities, enabling them to perceive, reason about, and imagine dynamic physical processes. Inspired by this capability, multimodal foundation models (MFMs) have emerged as powerful tools for multimodal understanding and generation. However, today's MFMs fall short of serving as effective world models. They lack the essential ability such as perform counterfactual reasoning, simulate dynamics, understand the spatiotemporal information, control generated visual outcomes, and perform multifaceted reasoning. We investigates what it takes to bridge the gap between multimodal foundation models and world models. We begin by improving the reasoning capabilities of MFMs through discriminative tasks and equipping MFMs with structured reasoning skills, such as causal inference, counterfactual thinking, and spatiotemporal reasoning, enabling them to go beyond surface correlations and understand deeper relationships within visual and textual data. Next, we explore generative capabilities of multimodal foundation models across both image and video modalities, introducing new frameworks for structured and controllable generation. Our approaches incorporate scene graphs, multimodal conditioning, and multimodal alignment strategies to guide the generation process, ensuring consistency with high-level semantics and fine-grained user intent. We further extend these techniques to controllable 4D generation, enabling interactive, editable, and morphable object synthesis over time and space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
