Synthesizing text-driven 3D human motion within realistic scenes requires learning both semantic intent ("walk to the couch") and physical feasibility (e.g., avoiding collisions). Current methods use generative frameworks that simultaneously learn high-level planning and low-level contact reasoning, and rely on computationally expensive 3D scene data such as point clouds or voxel occupancy grids. We propose SceMoS, a scene-aware motion synthesis framework that shows that structured 2D scene representations can serve as a powerful alternative to full 3D supervision in physically grounded motion synthesis. SceMoS disentangles global planning from local execution using lightweight 2D cues and relying on (1) a text-conditioned autoregressive global motion planner that operates on a bird's-eye-view (BEV) image rendered from an elevated corner of the scene, encoded with DINOv2 features, as the scene representation, and (2) a geometry-grounded motion tokenizer trained via a conditional VQ-VAE, that uses 2D local scene heightmap, thus embedding surface physics directly into a discrete vocabulary. This 2D factorization reaches an efficiency-fidelity trade-off: BEV semantics capture spatial layout and affordance for global reasoning, while local heightmaps enforce fine-grained physical adherence without full 3D volumetric reasoning. SceMoS achieves state-of-the-art motion realism and contact accuracy on the TRUMANS benchmark, reducing the number of trainable parameters for scene encoding by over 50%, showing that 2D scene cues can effectively ground 3D human-scene interaction.
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（北京大学计算机学院计算前沿研究中心）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；Jingdong Technology Information Technology Co., Ltd（京东科技信息技术有限公司）
Peking University（北京大学）；Carnegie Mellon University（卡内基梅隆大学）；Tongji University（同济大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Michigan（密歇根大学）
School of Information, University of Arizona（亚利桑那大学信息学院）
The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as "put a hot piece of bread on a plate". Currently, the best-performing models are able to complete less than 5% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to ge…
LLM-Grounded Dynamic Task Planning with Hierarchical Temporal Logic for Human-Aware Multi-Robot Handover
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人社会研究院）；Harbin Institute of Technology（哈尔滨工业大学）；The Chinese University of Hong Kong-Shenzhen（香港中文大学（深圳））；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）
VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, Agency for Science, Technology, and Research (A*STAR)（新加坡科技研究局（A*STAR）信息通信研究院）
SVLL: Staged Vision-Language Learning for Physically Grounded Embodied Task Planning
FNii-Shenzhen, CUHKSZ（FNii深圳，CUHKSZ）；SSE, CUHKSZ（SSE，CUHKSZ）；SAI, CUHKSZ（SAI，CUHKSZ）；Shenzhen University（深圳大学）；University of Sydney（悉尼大学）；Harbin Engineering University（哈尔滨工程大学）；Ising AI
SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models
Shanghai Innovation Institute（上海创新研究院）；OpenDriveLab at The University of Hong Kong（香港大学OpenDriveLab）；Li Auto Inc.（理想汽车）；Tongji University（同济大学）；Tsinghua University（清华大学）
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation. However, no prior dataset or benchmark has systematically evaluated physically grounded perception of complex social interactions that go beyond short actions, such as high-fiving, or simple group activities, such as gathering. In this work, we create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions by including social concepts such as helping another ag…
Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language to Pose (or JL2P), which learns a joint embedding…
