ESG：从文本描述生成物理一致的动态3D场景
ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions
浏览论文内容
中文总结 AI 辅助
提出基于演化场景图（ESG）的统一框架，利用大语言模型和可微仿真优化，从文本生成可直接在虚幻引擎中执行的物理一致动态3D场景，在事件完成度上显著优于现有基线。
中文摘要 AI 辅助
图像和3D场景生成领域的最新进展使得静态环境的生成越来越逼真，然而大多数方法仍局限于此类静态配置。从自然语言生成动态场景具有根本性的挑战：它需要对场景结构、时间演化和物理可行性进行联合推理，同时确保在现代物理引擎中的可靠执行。我们提出了一个统一框架，用于从文本生成物理一致的动态3D场景，其输出可直接在虚幻引擎中执行。我们方法的核心是“演化场景图”（ESG），它以机器可检查的形式指定具有物理属性、空间关系和事件驱动时间线的实体。给定提示词后，大型语言模型构建并验证完整的ESG；空间布局通过能量最小化的梯度优化进行落地；然后通过可微仿真优化受时间线约束的物理参数，以满足用户指定的事件；最终场景被编译为引擎可执行的类。在三个复杂度级别的10个场景上的实验表明，我们的方法实现了16.4/18的平均事件完成度，在事件完成度和参数准确性方面明显优于Scene Language、最强的引擎可执行基线（SimWorld）以及我们无物理优化的消融版本。
英文摘要
Recent progress in image and 3D scene generation has enabled increasingly realistic static environments, yet most methods remain confined to such static configurations. Generating dynamic scenes from natural language is fundamentally challenging: it requires joint reasoning over scene structure, temporal evolution, and physical feasibility, while ensuring reliable execution in modern physics engines. We present a unified framework for generating physically consistent dynamic 3D scenes from text, with outputs directly executable in Unreal Engine. Central to our approach is the \emph{Evolutive Scene Graph} (ESG), which specifies entities with physical attributes, spatial relations, and event-driven timelines in a machine-checkable form. Given a prompt, a large language model constructs and validates a complete ESG; spatial layouts are grounded via energy-minimized gradient optimization; timeline-constrained physical parameters are then optimized through differentiable simulation to satisfy user-specified events; and the resulting scene is compiled into an engine-executable class. Experiments on 10 scenes across three complexity levels show that our method achieves $16.4/18$ mean event completion, outperforming Scene Language, the strongest engine-executable baseline (SimWorld), and our ablation without physical optimization by a clear margin in event completion and parameter accuracy.
发表机构
- Zhejiang University（浙江大学）
- Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术研究院（KAIST））
- Northeastern University, China（中国东北大学）
机构由 AI 辅助整理，请以论文原文为准。
