Expressive Forecasting of 3D Whole-body Human Motions
MiLAB, Westlake University（西湖大学 MiLAB）；Zhejiang University（浙江大学）；Nanjing University of Science and Technology（南京理工大学）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；Xiaohongshu Inc.（小红书公司）
Pengxiang Ding, Qiongjie Cui, Min Zhang, Mengyuan Liu, Haofan Wang, Donglin Wang
人体运动预测旨在估计一段时间内的未来人体行为，是许多实际应用中的一项基础任务。然而，现有工作通常集中于预测人体的主要关节，而没有考虑人手的精细运动。在实际应用中，手势在人类与现实世界的交流中发挥着重要作用，并表达人类的主要意图。在本工作中，我们首次构建了全身人体姿态预测任务，联合预测未来的身体和手部活动。相应地，我们提出了一种新颖的编码-对齐-交互（Encoding-Alignment-Interaction，EAI）框架，旨在协同预测粗粒度（身体关节）和细粒度（手势）活动，从而实现具有表现力且相互促进的 3D 全身人体运动预测。具体而言，我们的模型包含两个关键组成部分：跨上下文对齐（cross-context alignment，XCA）和跨上下文交互（cross-context interaction，XCI）。考虑到全身内部的异构信息，XCA 旨在对齐不同人体组成部分的潜在特征，而 XCI 专注于有效捕捉人体组成部分之间的上下文交互。我们在一个新引入的大规模基准上进行了大量实验，并取得了最先进的性能。用于研究目的的代码已公开，地址为 https://github.com/Dingpx/EAI。
英文摘要：
Human motion forecasting, with the goal of estimating future human behavior over a period of time, is a fundamental task in many real-world applications. However, existing works typically concentrate on predicting the major joints of the human body without considering the delicate movements of the human hands. In practical applications, hand gesture plays an important role in human communication with the real world, and expresses the primary intention of human beings. In this work, we are the first to formulate a whole-body human pose forecasting task, which jointly predicts the future body and hand activities. Correspondingly, we propose a novel Encoding-Alignment-Interaction (EAI) framework that aims to predict both coarse (body joints) and fine-grained (gestures) activities collaboratively, enabling expressive and cross-facilitated forecasting of 3D whole-body human motions. Specifically, our model involves two key constituents: cross-context alignment (XCA) and cross-context interaction (XCI). Considering the heterogeneous information within the whole-body, XCA aims to align the latent features of various human components, while XCI focuses on effectively capturing the context interaction among the human components. We conduct extensive experiments on a newly-introduced large-scale benchmark and achieve state-of-the-art performance. The code is public for research purposes at https://github.com/Dingpx/EAI.
Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset
International Digital Economy Academy (IDEA)（国际数字经济学院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Recovering expressive humans from images is essential for understanding human behavior. Methods that estimate 3D bodies, faces, or hands have progressed significantly, yet separately. Face methods recover accurate 3D shape and geometric details, but need a tight crop and struggle with extreme views and low resolution. Whole-body methods are robust to a wide range of poses and resolutions, but provide only a rough 3D face shape without details like wrinkles. To get the best of both worlds, we introduce PIXIE, which produces animatable, whole-body 3D avatars with realistic facial detail, from a …
BrainCast: A Spatio-Temporal Forecasting Model for Whole-Brain fMRI Time Series Prediction
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究院）；MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（科学技术大学脑启发智能感知与认知教育部重点实验室）；School of Mathematics and Computer Sciences, Nanchang University（南昌大学数学与计算机科学学院）；School of Data Science, University of Virginia（弗吉尼亚大学数据科学学院）；Department of Biomedical Engineering, Tulane University（路易斯安那大学医学工程系）
AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation
SenseTime Research（商汤科技研究院）；City University of Hong Kong（香港城市大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab, Nanyang Technological University（新加坡南洋理工大学S-Lab）
Versatile Multimodal Controls for Expressive Talking Human Animation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University, Ant Group（人机混合增强智能国家级实验室，西安交通大学，蚂蚁集团）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi'an Jiaotong University（人机混合增强智能国家级实验室，西安交通大学）；University at Buffalo（布法罗大学）
Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset
International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Vision and Robotics, IDEA（国际数字经济学院计算机视觉与机器人系）；Whiting School of Engineering, Johns Hopkins University（约翰霍普金斯大学怀廷工程学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Multi-person pose forecasting remains a challenging problem, especially in modeling fine-grained human body interaction in complex crowd scenarios. Existing methods typically represent the whole pose sequence as a temporal series, yet overlook interactive influences among people based on skeletal body parts. In this paper, we propose a novel Trajectory-Aware Body Interaction Transformer (TBIFormer) for multi-person pose forecasting via effectively modeling body part interactions. Specifically, we construct a Temporal Body Partition Module that transforms all the pose sequences into a Multi-Per…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
The synthesis of human grasping has numerous applications including AR/VR, video games and robotics. While methods have been proposed to generate realistic hand-object interaction for object grasping and manipulation, these typically only consider interacting hand alone. Our goal is to synthesize whole-body grasping motions. Starting from an arbitrary initial pose, we aim to generate diverse and natural whole-body human motions to approach and grasp a target object in 3D space. This task is challenging as it requires modeling both whole-body dynamics and dexterous finger movements. To this end…
Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components t…
Autonomous driving vehicles (ADVs) hold great hopes to solve traffic congestion problems and reduce the number of traffic accidents. Accurate trajectories prediction of other traffic agents around ADVs is of key importance to achieve safe and efficient driving. Pedestrians, particularly, are more challenging to forecast due to their complex social in-teractions and randomly moving patterns. We propose a Residual Graph Convolutional Neural Network (Res-GCNN), which models the interactive behaviors of pedes-trians by using the adjacent matrix of the constructed graph for the current scene. Thoug…
We propose a 3D face generative model with local weights to increase the model's variations and expressiveness. The proposed model allows partial manipulation of the face while still learning the whole face mesh. For this purpose, we address an effective way to extract local facial features from the entire data and explore a way to manipulate them during a holistic generation. First, we factorize the latent space of the whole face to the subspace indicating different parts of the face. In addition, local weights generated by non-negative matrix factorization are applied to the factorized laten…
Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Paradigm（第四范式）；Université Paris-Saclay（巴黎萨克雷大学）；ChaLearn
Human behavior forecasting during human-human interactions is of utmost importance to provide robotic or virtual agents with social intelligence. This problem is especially challenging for scenarios that are highly driven by interpersonal dynamics. In this work, we present the first systematic comparison of state-of-the-art approaches for behavior forecasting. To do so, we leverage whole-body annotations (face, body, and hands) from the very recently released UDIVA v0.5, which features face-to-face dyadic interactions. Our best attention-based approaches achieve state-of-the-art performance in…
