We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We find that RELATE is also amenable to physically realistic scene editing and that it significantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (cars). In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity. Source code, datasets and more results are available at http://geometry.cs.ucl.ac.uk/projects/2020/relate/.
FlowHMR: Physically Plausible Motion Capture from Video
Peking University（北京大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tencent（腾讯）；East China Normal University（华东师范大学）；Sun Yat-sen University（中山大学）；Zhejiang University（浙江大学）
Chain of Event-Centric Causal Thought for Physically Plausible Video Generation
Sichuan University（四川大学）；The University of Adelaide（阿德莱德大学）；Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）
Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioning
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Computer Science, University of Adelaide（阿德莱德大学计算机学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction
Southern University of Science and Technology（南方科技大学）；Spatialtemporal AI（时空人工智能）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；University of Sheffield（谢菲尔德大学）
VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior
Monash University（莫纳什大学）；Dalian University of Technology（大连理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Oxford University（牛津大学）；The University of Sydney（悉尼大学）；ZMO AI（智源人工智能）
MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；StepFun（阶跃星辰）；Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information（多模态信息超级智能安全北京市重点实验室）；School of Information Science and Technology, Shanghai Tech University（上海科技大学信息科学与技术学院）
OrthoPhys: Physically Plausible Video Generation with Orthogonal-View Geometry Guidance
the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；the School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；College of Automotive and Energy Engineering, Tongji University（同济大学汽车与能源工程学院）；SKLCCSE, School of Computer Science and Engineering, Beihang University（SKLCCSE，北京航空航天大学计算机科学与工程学院）
SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models
School of Computer Science, Wuhan University（武汉大学计算机学院）；Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
PhysPlan: Grounded Physical State Reasoning and Graph-Guided Optimization for Physically Plausible Video Generation
University of Science, Ho Chi Minh City（胡志明市理科大学）；Vietnam National University, Ho Chi Minh City（越南国立大学胡志明市分校）；Monash University（莫纳什大学）；University of Dayton（代顿大学）
George Mason University（乔治梅森大学）；Adobe Research（Adobe研究院）；NVIDIA（英伟达）
The ability to edit materials of objects in images is desirable by many content creators. However, this is an extremely challenging task as it requires to disentangle intrinsic physical properties of an image. We propose an end-to-end network architecture that replicates the forward image formation process to accomplish this task. Specifically, given a single image, the network first predicts intrinsic properties, i.e. shape, illumination, and material, which are then provided to a rendering layer. This layer performs in-network image synthesis, thereby enabling the network to understand the p…
PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimization
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS（LIGM，巴黎理工学院，IP巴黎，古斯塔夫·埃菲尔大学，国家科学研究中心）；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
CP4D: Compositional Physics-aware 4D Scene Generation
School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）；Zhongguancun Academy（中关村学院）；the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Eastern Institute of Technology（东部技术研究所）；Tsinghua University（清华大学）
Reconstructing the missing parts of a curve has been the subject of much computational research, with applications in image inpainting, object synthesis, etc. Different approaches for solving that problem are typically based on processes that seek visually pleasing or perceptually plausible completions. In this work we focus on reconstructing the underlying physically likely shape by utilizing the global statistics of natural curves. More specifically, we develop a reconstruction model that seeks the mean physical curve for a given inducer configuration. This simple model is both straightforwa…
Due to the visual ambiguity, purely kinematic formulations on monocular human motion capture are often physically incorrect, biomechanically implausible, and can not reconstruct accurate interactions. In this work, we focus on exploiting the high-precision and non-differentiable physics simulator to incorporate dynamical constraints in motion capture. Our key-idea is to use real physical supervisions to train a target pose distribution prior for sampling-based motion control to capture physically plausible human motion. To obtain accurate reference motion with terrain interactions for the samp…
PAMD: Plausibility-Aware Motion Diffusion Model for Long Dance Generation
School of Computer Science and Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Southeast University（计算机科学与工程学院，新一代人工智能技术及交叉应用重点实验室，教育部，东南大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；School of Computer Science and Software Engineering, National Engineering Laboratory for Big Data System Computing Technology, Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（计算机科学与软件工程学院，大数据系统计算技术国家工程实验室，广东智能信息处理重点实验室，深圳大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
School of AIA, Huazhong University of Science and Technology（华中科技大学人工智能学院）；the S-Lab, Nanyang Technological University (NTU)（南洋理工大学S实验室）；the Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；Zhejiang University (ZJU)（浙江大学）；the Institute for AI Industry Research (AIR), Tsinghua University (THU)（清华大学人工智能产业研究院）
Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields
Institute for AI, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理学与认知科学学院）；School of EECS, Peking University（北京大学电子工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Yuanpei College, Peking University（北京大学元培学院）；State Key Lab of General AI, Peking University（北京大学通用人工智能国家重点实验室）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室）
PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
University of Tübingen, Zuse School ELIZA（图宾根大学Zuse学校ELIZA）；University of Tübingen, Tübingen AI Center（图宾根大学图宾根人工智能中心）；University of Tübingen, Tübingen AI Center, MPI for Informatics, SIC（图宾根大学图宾根人工智能中心、马克斯·普朗克信息研究所、SIC）
All hand-object interaction is controlled by forces that the two bodies exert on each other, but little work has been done in modeling these underlying forces when doing pose and contact estimation from RGB/RGB-D data. Given the pose of the hand and object from any pose estimation system, we propose an end-to-end differentiable model that refines pose estimates by learning the forces experienced by the object at each vertex in its mesh. By matching the learned net force to an estimate of net force based on finite differences of position, this model is able to find forces that accurately descri…
Recently Convolutional Neural Networks (CNN) have been used to reconstruct hyperspectral information from RGB images. Moreover, this spectral reconstruction problem (SR) can often be solved with good (low) error. However, these methods are not physically plausible: that is when the recovered spectra are reintegrated with the underlying camera sensitivities, the resulting predicted RGB is not the same as the actual RGB, and sometimes this discrepancy can be large. The problem is further compounded by exposure change. Indeed, most learning-based SR models train for a fixed exposure setting and w…
