Video generation has achieved impressive quality, but it still suffers from artifacts such as temporal inconsistency and violation of physical laws. Leveraging 3D scenes can fundamentally resolve these issues by providing precise control over scene entities. To facilitate the easy generation of diverse photorealistic scenes, we propose Scene Copilot, a framework combining large language models (LLMs) with a procedural 3D scene generator. Specifically, Scene Copilot consists of Scene Codex, BlenderGPT, and Human in the loop. Scene Codex is designed to translate textual user input into commands understandable by the 3D scene generator. BlenderGPT provides users with an intuitive and direct way to precisely control the generated 3D scene and the final output video. Furthermore, users can utilize Blender UI to receive instant visual feedback. Additionally, we have curated a procedural dataset of objects in code format to further enhance our system's capabilities. Each component works seamlessly together to support users in generating desired 3D scenes. Extensive experiments demonstrate the capability of our framework in customizing 3D scenes and video generation.
SceneX: Procedural Controllable Large-scale Scene Generation
University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics（人工智能与机器人研究中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shandong University（山东大学）；University of Science and Technology Beijing（北京科技大学）；China University of Geosciences Beijing（中国地质大学（北京））
For low-level computer vision and image processing ML tasks, training on large datasets is critical for generalization. However, the standard practice of relying on real-world images primarily from the Internet comes with image quality, scalability, and privacy issues, especially in commercial contexts. To address this, we have developed a procedural synthetic data generation pipeline and dataset tailored to low-level vision tasks. Our Unreal engine-based synthetic data pipeline populates large scenes algorithmically with a combination of random 3D objects, materials, and geometric transformat…
University of Konstanz（康斯坦茨大学）；Google Brain（谷歌大脑）；Purdue University（普渡大学）
The placement of vegetation plays a central role in the realism of virtual scenes. We introduce procedural placement models (PPMs) for vegetation in urban layouts. PPMs are environmentally sensitive to city geometry and allow identifying plausible plant positions based on structural and functional zones in an urban layout. PPMs can either be directly used by defining their parameters or can be learned from satellite images and land register data. Together with approaches for generating buildings and trees, this allows us to populate urban landscapes with complex 3D vegetation. The effectivenes…
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We present a novel procedural framework to generate an arbitrary number of labeled crowd videos (LCrowdV). The resulting crowd video datasets are used to design accurate algorithms or training models for crowded scene understanding. Our overall approach is composed of two components: a procedural simulation framework for generating crowd movements and behaviors, and a procedural rendering framework to generate different videos or images. Each video or image is automatically labeled based on the environment, number of pedestrians, density, behavior, flow, lighting conditions, viewpoint, noise, …
SenseTime Group Limited（商汤集团有限公司）；Department of Information Engineering, the Chinese University of Hong Kong（香港中文大学信息工程系）
Over the past few years there is a growing interest in the learning-based self driving system. To ensure safety, such systems are first developed and validated in simulators before being deployed in the real world. However, most of the existing driving simulators only contain a fixed set of scenes and a limited number of configurable settings. That might easily cause the overfitting issue for the learning-based driving systems as well as the lack of their generalization ability to unseen scenarios. To better evaluate and improve the generalization of end-to-end driving, we introduce an open-en…
Several families of continual learning techniques have been proposed to alleviate catastrophic interference in deep neural network training on non-stationary data. However, a comprehensive comparison and analysis of limitations remains largely open due to the inaccessibility to suitable datasets. Empirical examination not only varies immensely between individual works, it further currently relies on contrived composition of benchmarks through subdivision and concatenation of various prevalent static vision datasets. In this work, our goal is to bridge this gap by introducing a computer graphic…
NVIDIA（英伟达）；University of Toronto（多伦多大学）；University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. In order to produce realistic and diverse scenes, a number of parameters governing the procedural models have to be carefully tuned by experts. These parameters control both the structure of scenes being generated (e.g. how many cars in the scene), as well as parameters which place objects in valid configurations. Meta-Sim aimed at automatically tuning parameters given a target collection of real images in an unsupervised way. In Meta-Sim2, we aim to learn the s…
A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation
Antal Bejczy Center for Intelligent Robotics（安塔尔·贝茨奇智能机器人中心）；Research and Innovation Center of Obuda University（奥布达大学研究与创新中心）
SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；IRCCS Humanitas Research Hospital（IRCCS人类itas研究医院）；UCL Hawkes Institute and Department of Computer Science（UCL Hawkes研究所和计算机科学系）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）
CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
University of Chinese Academy of Sciences（中国科学院大学）；China University of Geosciences (Beijing)（中国地质大学（北京））；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Shandong University（山东大学）；University of Science and Technology Beijing（北京科技大学）
To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoint, as the two variables jointly give rise to the agent's observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of lat…
Samsung AI Center NY（三星纽约人工智能中心）；Columbia University（哥伦比亚大学）
Studies in robot teleoperation have been centered around action specifications -- from continuous joint control to discrete end-effector pose control. However, these robot-centric interfaces often require skilled operators with extensive robotics expertise. To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of …
LLM-Driven 3D Scene Generation of Agricultural Simulation Environments
Faculty of Applied Engineering – Dept. of Electronics and ICT, University of Antwerp（应用工程学院——电子与信息通信技术系，安特卫普大学）；Flanders Make Strategic Research Centre（弗拉芒制造战略研究中心）
