We introduce PerpetualWonder, a hybrid generative simulator that enables long-horizon, action-conditioned 4D scene generation from a single image. Current works fail at this task because their physical state is decoupled from their visual representation, which prevents generative refinements to update the underlying physics for subsequent interactions. PerpetualWonder solves this by introducing the first true closed-loop system. It features a novel unified representation that creates a bidirectional link between the physical state and visual primitives, allowing generative refinements to correct both the dynamics and appearance. It also introduces a robust update mechanism that gathers supervision from multiple viewpoints to resolve optimization ambiguity. Experiments demonstrate that from a single image, PerpetualWonder can successfully simulate complex, multi-step interactions from long-horizon actions, maintaining physical plausibility and visual consistency.
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Huawei Cloud & AI（华为云与计算BG）
A bathtub in a library, a sink in an office, a bed in a laundry room -- the counter-intuition suggests that scene provides important prior knowledge for 3D object detection, which instructs to eliminate the ambiguous detection of similar objects. In this paper, we propose HyperDet3D to explore scene-conditioned prior knowledge for 3D object detection. Existing methods strive for better representation of local elements and their relations without scene-conditioned knowledge, which may cause ambiguity merely based on the understanding of individual points and object candidates. Instead, HyperDet…
Beijing Advanced Innovation Center for Intelligent Robots and Systems（北京智能机器人与系统高精尖创新中心）；PEK-SenseTime Joint Laboratory of Machine Vision（北京大学-商汤科技机器视觉联合实验室）
Semantic Scene Completion (SSC) aims to simultaneously predict the volumetric occupancy and semantic category of a 3D scene. In this paper, we propose a real-time semantic scene completion method with a feature aggregation strategy and conditioned prediction module. Feature aggregation fuses feature with different receptive fields and gathers context to improve scene completion performance. And the conditioned prediction module adopts a two-step prediction scheme that takes volumetric occupancy as a condition to enhance semantic completion prediction. We conduct experiments on three recognized…
Designing complex 3D scenes has been a tedious, manual process requiring domain expertise. Emerging text-to-3D generative models show great promise for making this task more intuitive, but existing approaches are limited to object-level generation. We introduce \textbf{locally conditioned diffusion} as an approach to compositional scene diffusion, providing control over semantic parts using text prompts and bounding boxes while ensuring seamless transitions between these parts. We demonstrate a score distillation sampling--based text-to-3D synthesis pipeline that enables compositional 3D scene…
Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching
The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）；Tianfu Jiangxi Laboratory（天斧江西实验室）；Monash University（墨尔本大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；University of Chinese Academy of Sciences（中国科学院大学）
Scene completion refers to obtaining dense scene representation from an incomplete perception of complex 3D scenes. This helps robots detect multi-scale obstacles and analyse object occlusions in scenarios such as autonomous driving. Recent advances show that implicit representation learning can be leveraged for continuous scene completion and achieved through physical constraints like Eikonal equations. However, former Eikonal completion methods only demonstrate results on watertight meshes at a scale of tens of meshes. None of them are successfully done for non-watertight LiDAR point clouds …
Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability
Technische Universität Berlin（柏林工业大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔国立综合理工学院、法国国家科学研究中心格勒诺布尔计算机科学实验室）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）；Science of Intelligence Research Cluster of Excellence（智能科学卓越研究集群）
SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
The School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Waytous Inc.（Waytous公司）；The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）
Beijing Institute of Technology（北京理工大学）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；Peking University（北京大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit o…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；Centre of Perceptual and Interactive Intelligence（感知与交互智能研究中心）
We revisit human motion synthesis, a task useful in various real world applications, in this paper. Whereas a number of methods have been developed previously for this task, they are often limited in two aspects: focusing on the poses while leaving the location movement behind, and ignoring the impact of the environment on the human motion. In this paper, we propose a new framework, with the interaction between the scene and the human motion taken into account. Considering the uncertainty of human motion, we formulate this task as a generative task, whose objective is to generate plausible hum…
National Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能国家重点实验室）；Tsinghua University（清华大学）；Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）
We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior works, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion. Such a design alleviates the discrepancies among different modules and…
Apple（苹果公司）；University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）
We tackle the challenge of learning a distribution over complex, realistic, indoor scenes. In this paper, we introduce Generative Scene Networks (GSN), which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera. Our model can be used as a prior to generate new scenes, or to complete a scene given only sparse 2D observations. Recent work has shown that generative models of radiance fields can capture properties such as multi-view consistency and view-dependent lighting. However, these models are specialized for constrained vie…
FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow
School of Computer Science, Peking University（北京大学计算机科学系）；Technical University of Munich（慕尼黑技术大学）；Beijing Jiaotong University（北京交通大学）；Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究院）；Theta Labs, Inc.（Theta Labs公司）；Beijing Normal University（北京师范大学）
What does human pose tell us about a scene? We propose a task to answer this question: given human pose as input, hallucinate a compatible scene. Subtle cues captured by human pose -- action semantics, environment affordances, object interactions -- provide surprising insight into which scenes are compatible. We present a large-scale generative adversarial network for pose-conditioned scene generation. We significantly scale the size and complexity of training data, curating a massive meta-dataset containing over 19 million frames of humans in everyday environments. We double the capacity of o…
We address the task of indoor scene generation by generating a sequence of objects, along with their locations and orientations conditioned on a room layout. Large-scale indoor scene datasets allow us to extract patterns from user-designed indoor scenes, and generate new scenes based on these patterns. Existing methods rely on the 2D or 3D appearance of these scenes in addition to object positions, and make assumptions about the possible relations between objects. In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of …
T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation
School of Computer Science and Engineering（计算机科学与工程学院）；Nanjing University of Science and Technology（南京理工大学）；Faculty of Data Science（数据科学学院）；City University of Macau（澳门城市大学）；Beijing Institute of Technology（北京理工大学）；Fondazione Bruno Kessler（布鲁诺·凯瑟尔基金会）；School of Public Health（公共卫生学院）；Shanghai Jiao Tong University（上海交通大学）；School of Computing（计算机学院）；National University of Singapore（新加坡国立大学）
We address the problem of forecasting pedestrian and vehicle trajectories in unknown environments, conditioned on their past motion and scene structure. Trajectory forecasting is a challenging problem due to the large variation in scene structure and the multimodal distribution of future trajectories. Unlike prior approaches that directly learn one-to-many mappings from observed context to multiple future trajectories, we propose to condition trajectory forecasts on plans sampled from a grid based policy learned using maximum entropy inverse reinforcement learning (MaxEnt IRL). We reformulate …
VoxScene: Anchor-Conditioned Voxel Diffusion for Indoor Scene Arrangement
Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；Peking University（北京大学）
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；Snap Inc.（Snap公司）；CUHK（香港中文大学）；Stanford（斯坦福大学）；NTU（南洋理工大学）；HKU（香港大学）；University of Michigan（密歇根大学）；SFU（西蒙菲莎大学）；Google DeepMind（谷歌DeepMind）
RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Columbia University（哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon（亚马逊公司）
Controllable Image Captioning with Prompt-Conditioned Scene Rewards
POSTECH（浦项科技大学）；Graduate School of Artificial Intelligence, POSTECH（浦项科技大学人工智能研究生院）；Department of Computer Science and Engineering, POSTECH（浦项科技大学计算机科学与工程系）
Generating images from semantic visual knowledge is a challenging task, that can be useful to condition the synthesis process in complex, subtle, and unambiguous ways, compared to alternatives such as class labels or text descriptions. Although generative methods conditioned by semantic representations exist, they do not provide a way to control the generation process aside from the specification of constraints between objects. As an example, the possibility to iteratively generate or modify images by manually adding specific items is a desired property that, to our knowledge, has not been ful…
