基于空间知识图的多模态合成
Spatial Knowledge Graph-Guided Multimodal Synthesis
AI总结：
SKG2DATA通过空间知识图引导多模态合成，提升多模态大语言模型的空间感知与推理能力。
摘要：
最近多模态大语言模型（MLLMs）的进步显著增强了其能力；然而，其空间感知能力仍是一个显著的限制。为解决这一挑战，多模态数据合成提供了一个有希望的解决方案。然而，确保合成的数据符合空间常识是一项非同寻常的任务。我们的方法通过提供一个系统框架来生成空间上一致的数据来解决这一关键缺口。在本工作中，我们引入了SKG2DATA，一种新的多模态合成方法，由空间知识图引导，基于知识到数据生成的概念。SKG2DATA采用自动化流程构建空间知识图（SKG），有效捕捉人类样的空间认知，包括方向和距离关系。这些结构化表示随后作为精确指导我们的集成合成流程，其中扩散模型生成空间一致的图像，而MLLM生成相应的文本描述。自动构建SKG使能够可扩展地生成多样化且真实的空间配置，克服了手动数据收集和标注的限制。广泛的实验表明，从不同类型的空间知识，包括方向和距离，合成的数据显著增强了MLLM的空间感知和推理能力，尽管略微牺牲了其通用能力。我们希望知识驱动的数据合成的想法能促进空间智能的发展。代码可在https://github.com/zjunlp/Knowledge2Data获得。
英文摘要：
Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities; however, their spatial perception abilities remain a notable limitation. To address this challenge, multimodal data synthesis offers a promising solution. Yet, ensuring that synthesized data adhere to spatial common sense is a non-trivial task. Our approach addresses this critical gap by providing a systematic framework for generating spatially coherent data. In this work, we introduce SKG2DATA, a novel multimodal synthesis approach guided by spatial knowledge graphs, grounded in the concept of knowledge-to-data generation. SKG2DATA employs an automated pipeline for constructing Spatial Knowledge Graph (SKG) that effectively captures human-like spatial cognition, including directional and distance relationships. These structured representations then serve as precise guidance for our integrated synthesis pipeline, where a diffusion model generates spatially-consistent images while a MLLM produces corresponding textual descriptions. The automated construction of SKG enables scalable generation of diverse yet realistic spatial configurations, overcoming the limitations of manual data collection and annotation. Extensive experiments demonstrate that data synthesized from diverse types of spatial knowledge, including direction and distance, enhance the spatial perception and reasoning abilities of MLLMs markedly, albeit with a slight cost to their general capabilities. We hope that the idea of knowledge-based data synthesis can advance the development of spatial intelligence. Code is available at https://github.com/zjunlp/Knowledge2Data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
