Human motion prediction, i.e., forecasting future body poses given observed pose sequence, has typically been tackled with recurrent neural networks (RNNs). However, as evidenced by prior work, the resulted RNN models suffer from prediction errors accumulation, leading to undesired discontinuities in motion prediction. In this paper, we propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. In this context, we then propose to encode temporal information by working in trajectory space, instead of the traditionally-used pose space. This alleviates us from manually defining the range of temporal dependencies (or temporal convolutional filter size, as done in previous work). Moreover, spatial dependency of human pose is encoded by treating a human pose as a generic graph (rather than a human skeletal kinematic tree) formed by links between every pair of body joints. Instead of using a pre-defined graph structure, we design a new graph convolutional network to learn graph connectivity automatically. This allows the network to capture long range dependencies beyond that of human kinematic tree. We evaluate our approach on several standard benchmark datasets for motion prediction, including Human3.6M, the CMU motion capture dataset and 3DPW. Our experiments clearly demonstrate that the proposed approach achieves state of the art performance, and is applicable to both angle-based and position-based pose representations. The code is available at https://github.com/wei-mao-2019/LearnTrajDep
Human motion prediction from motion capture data is a classical problem in the computer vision, and conventional methods take the holistic human body as input. These methods ignore the fact that, in various human activities, different body components (limbs and the torso) have distinctive characteristics in terms of the moving pattern. In this paper, we argue local representations on different body components should be learned separately and, based on such idea, propose a network, Skeleton Network (SkelNet), for long-term human motion prediction. Specifically, at each time-step, local structur…
Beijing University of Posts and Telecommunications（北京邮电大学）
Human motion prediction is an essential part for human-robot collaboration. Unlike most of the existing methods mainly focusing on improving the effectiveness of spatiotemporal modeling for accurate prediction, we take effectiveness and efficiency into consideration, aiming at the prediction quality, computational efficiency and the lightweight of the model. A multi-grained trajectory graph convolutional networks based and lightweight framework is proposed for habit-unrelated human motion prediction. Specifically, we represent human motion as multi-grained trajectories, including joint traject…
Trajectory forecasting plays a pivotal role in the field of intelligent vehicles or social robots. Recent works focus on modeling spatial social impacts or temporal motion attentions, but neglect inherent properties of motions, i.e. moving trends and driving intentions. This paper proposes a context-free Hierarchical Motion Encoder-Decoder Network (HMNet) for vehicle trajectory prediction. HMNet first infers the hierarchical difference on motions to encode physically compliant patterns with high expressivity of moving trends and driving intentions. Then, a goal (endpoint)-embedded decoder hier…
University of Central Florida（中佛罗里达大学）；Purdue University（普渡大学）
Mining the underlying patterns in gigantic and complex data is of great importance to data analysts. In this paper, we propose a motion pattern approach to mine frequent behaviors in trajectory data. Motion patterns, defined by a set of highly similar flow vector groups in a spatial locality, have been shown to be very effective in extracting dominant motion behaviors in video sequences. Inspired by applications and properties of motion patterns, we have designed a framework that successfully solves the general task of trajectory clustering. Our proposed algorithm consists of four phases: flow…
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Predicting future human motion is critical for intelligent robots to interact with humans in the real world, and human motion has the nature of multi-granularity. However, most of the existing work either implicitly modeled multi-granularity information via fixed modes or focused on modeling a single granularity, making it hard to well capture this nature for accurate predictions. In contrast, we propose a novel end-to-end network, Semi-Decoupled Multi-grained Trajectory Learning network (SDMTL), to predict future poses, which not only flexibly captures rich multi-grained trajectory informatio…
Forecasting human trajectories is critical for tasks such as robot crowd navigation and autonomous driving. Modeling social interactions is of great importance for accurate group-wise motion prediction. However, most existing methods do not consider information about coherence within the crowd, but rather only pairwise interactions. In this work, we propose a novel framework, coherent motion aware graph convolutional network (CoMoGCN), for trajectory prediction in crowded scenes with group constraints. First, we cluster pedestrian trajectories into groups according to motion coherence. Then, w…
SenseTime Group Limited（商汤集团有限公司）；The Chinese University of Hong Kong（香港中文大学）
Making accurate motion prediction of the surrounding traffic agents such as pedestrians, vehicles, and cyclists is crucial for autonomous driving. Recent data-driven motion prediction methods have attempted to learn to directly regress the exact future position or its distribution from massive amount of trajectory data. However, it remains difficult for these methods to provide multimodal predictions as well as integrate physical constraints such as traffic rules and movable areas. In this work we propose a novel two-stage motion prediction framework, Trajectory Proposal Network (TPNet). TPNet…
MoTDiff: High-resolution Motion Trajectory estimation from a single blurred image using Diffusion models
Department of Artificial Intelligence (AI), Sungkyunkwan University (SKKU)（人工智能系，全北国立大学）；Department of Electrical and Computer Engineering (ECE), SKKU（电气电子工程系，全北国立大学）；New Technology Team, ALLforLAND Co., Ltd.（ALLforLAND公司新技术团队）；Department of ECE, SKKU（电气电子工程系，全北国立大学）；Departments of AI, ECE, Semiconductor Convergence Engineering, Advanced Display Engineering, and Display Convergence Engineering, SKKU（人工智能系、电气电子工程系、半导体融合工程系、先进显示工程系和显示融合工程系，全北国立大学）；Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon（神经科学成像研究中心，基础科学研究所）
Xi’an Jiaotong University（西安交通大学）；Chang’an University（长安大学）；The University of Sydney（悉尼大学）
Trajectory forecasting is critical for autonomous platforms to make safe planning and actions. Currently, most trajectory forecasting methods assume that object trajectories have been extracted and directly develop trajectory predictors based on the ground truth trajectories. However, this assumption does not hold in practical situations. Trajectories obtained from object detection and tracking are inevitably noisy, which could cause serious forecasting errors to predictors built on ground truth trajectories. In this paper, we propose a trajectory predictor directly based on detection results …
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）
Synthesis of long-term human motion skeleton sequences is essential to aid human-centric video generation with potential applications in Augmented Reality, 3D character animations, pedestrian trajectory prediction, etc. Long-term human motion synthesis is a challenging task due to multiple factors like, long-term temporal dependencies among poses, cyclic repetition across poses, bi-directional and multi-scale dependencies among poses, variable speed of actions, and a large as well as partially overlapping space of temporal pose variations across multiple class/types of human activities. This p…
Uncovering the human motion pattern: Pattern Memory-based Diffusion Model for Trajectory Prediction
Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia（智能电信软件与多媒体北京市重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）
Predicting motion of surrounding agents is critical to real-world applications of tactical path planning for autonomous driving. Due to the complex temporal dependencies and social interactions of agents, on-line trajectory prediction is a challenging task. With the development of attention mechanism in recent years, transformer model has been applied in natural language sequence processing first and then image processing. In this paper, we present a Spatial-Channel Transformer Network for trajectory prediction with attention functions. Instead of RNN models, we employ transformer model to cap…
This paper introduces a joint learning architecture (JLA) for multiple object tracking (MOT) and trajectory forecasting in which the goal is to predict objects' current and future trajectories simultaneously. Motion prediction is widely used in several state of the art MOT methods to refine predictions in the form of bounding boxes. Typically, a Kalman Filter provides short-term estimations to help trackers correctly predict objects' locations in the current frame. However, the Kalman Filter-based approaches cannot predict non-linear trajectories. We propose to jointly train a tracking and tra…
This paper revisits the problem of continuous-time structure from motion, and introduces a number of extensions that improve convergence and efficiency. The formulation with a $\mathcal{C}^2$-continuous spline for the trajectory naturally incorporates inertial measurements, as derivatives of the sought trajectory. We analyse the behaviour of split interpolation on $\mathbb{SO}(3)$ and on $\mathbb{R}^3$, and a joint interpolation on $\mathbb{SE}(3)$, and show that the latter implicitly couples the direction of translation and rotation. Such an assumption can make good sense for a camera mounted…
Nanjing University of Science and Technology（南京理工大学）；Yale-NUS College（耶鲁-新加坡国立大学学院）；National University of Singapore（新加坡国立大学）
Occlusion is a long-standing problem that causes many modern tracking methods to be erroneous. In this paper, we address the occlusion problem by exploiting the current and future possible locations of the target object from its past trajectory. To achieve this, we introduce a learning-based tracking method that takes into account background motion modeling and trajectory prediction. Our trajectory prediction module predicts the target object's locations in the current and future frames based on the object's past trajectory. Since, in the input video, the target object's trajectory is not only…
Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）
Learning the spatial-temporal representation of motion information is crucial to human action recognition. Nevertheless, most of the existing features or descriptors cannot capture motion information effectively, especially for long-term motion. To address this problem, this paper proposes a long-term motion descriptor called sequential Deep Trajectory Descriptor (sDTD). Specifically, we project dense trajectories into two-dimensional planes, and subsequently a CNN-RNN network is employed to learn an effective representation for long-term motion. Unlike the popular two-stream ConvNets, the sDT…
SCT-MOT: Enhancing Air-to-Air Multiple UAVs Tracking with Swarm-Coupled Motion and Trajectory Guidance
China-UAE Belt and Road Joint Laboratory on Intelligent Unmanned Systems of School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学宇航学院中国-阿联酋‘一带一路’智能无人系统联合实验室）；School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学宇航学院）；School of Information and Computing Science, Xi'an Jiaotong-Liverpool University（西交利物浦大学信息与计算科学学院）
Progressive Pretext Task Learning for Human Trajectory Prediction
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）
Beijing University of Posts and Telecommunications（北京邮电大学）；City University of Hong Kong（香港城市大学）；Tsinghua University（清华大学）
Human motion prediction is an increasingly interesting topic in computer vision and robotics. In this paper, we propose a new 2D CNN based network, TrajectoryNet, to predict future poses in the trajectory space. Compared with most existing methods, our model focuses on modeling the motion dynamics with coupled spatio-temporal features, local-global spatial features and global temporal co-occurrence features of the previous pose sequence. Specifically, the coupled spatio-temporal features describe the spatial and temporal structure information hidden in the natural human motion sequence, which …
In the domain of sequence modelling, Recurrent Neural Networks (RNN) have been capable of achieving impressive results in a variety of application areas including visual question answering, part-of-speech tagging and machine translation. However this success in modelling short term dependencies has not successfully transitioned to application areas such as trajectory prediction, which require capturing both short term and long term relationships. In this paper, we propose a Tree Memory Network (TMN) for modelling long term and short term relationships in sequence-to-sequence mapping problems. …
University of Sydney（悉尼大学）；Birkbeck College（伯贝克学院）
Motion blur in dynamic scenes is an important yet challenging research topic. Recently, deep learning methods have achieved impressive performance for dynamic scene deblurring. However, the motion information contained in a blurry image has yet to be fully explored and accurately formulated because: (i) the ground truth of dynamic motion is difficult to obtain; (ii) the temporal ordering is destroyed during the exposure; and (iii) the motion estimation from a blurry image is highly ill-posed. By revisiting the principle of camera exposure, motion blur can be described by the relative motions o…
Human trajectory forecasting in crowds, at its core, is a sequence prediction problem with specific challenges of capturing inter-sequence dependencies (social interactions) and consequently predicting socially-compliant multimodal distributions. In recent years, neural network-based methods have been shown to outperform hand-crafted methods on distance-based metrics. However, these data-driven methods still suffer from one crucial limitation: lack of interpretability. To overcome this limitation, we leverage the power of discrete choice models to learn interpretable rule-based intents, and su…
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；Snap Inc.（Snap公司）；CUHK（香港中文大学）；Stanford（斯坦福大学）；NTU（南洋理工大学）；HKU（香港大学）；University of Michigan（密歇根大学）；SFU（西蒙菲莎大学）；Google DeepMind（谷歌DeepMind）
Trajectory prediction of road users in real-world scenarios is challenging because their movement patterns are stochastic and complex. Previous pedestrian-oriented works have been successful in modelling the complex interactions among pedestrians, but fail in predicting trajectories when other types of road users are involved (e.g., cars, cyclists, etc.), because they ignore user types. Although a few recent works construct densely connected graphs with user label information, they suffer from superfluous spatial interactions and temporal dependencies. To address these issues, we propose Multi…
Learning Cooperative Trajectory Representations for Motion Forecasting
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Chinese Academy of Science（中国科学院大学）；The University of Hong Kong（香港大学）
Xi’an Jiaotong University（西安交通大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Research（微软研究院）
Video frame interpolation (VFI) aims to synthesize an intermediate frame between two consecutive frames. State-of-the-art approaches usually adopt a two-step solution, which includes 1) generating locally-warped pixels by flow-based motion estimations, 2) blending the warped pixels to form a full frame through deep neural synthesis networks. However, due to the inconsistent warping from the two consecutive frames, the warped features for new frames are usually not aligned, which leads to distorted and blurred frames, especially when large and complex motions occur. To solve this issue, in this…
Intention Enhanced Diffusion Model for Multimodal Pedestrian Trajectory Prediction
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, the Southern University of Science and Technology, Shenzhen 518055, China（广东省全自动化系统控制理论与技术重点实验室，南方科技大学，深圳518055，中国）；Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China（香港城市大学机械工程系，香港特别行政区，中国）
SenseTime Research（商汤科技研究院）；National University of Singapore（新加坡国立大学）
Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. Th…
We present GTT-Net, a supervised learning framework for the reconstruction of sparse dynamic 3D geometry. We build on a graph-theoretic formulation of the generalized trajectory triangulation problem, where non-concurrent multi-view imaging geometry is known but global image sequencing is not provided. GTT-Net learns pairwise affinities modeling the spatio-temporal relationships among our input observations and leverages them to determine 3D geometry estimates. Experiments reconstructing 3D motion-capture sequences show GTT-Net outperforms the state of the art in terms of accuracy and robustne…
University of California, Los Angeles（加州大学洛杉矶分校）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）
Human behavior has the nature of indeterminacy, which requires the pedestrian trajectory prediction system to model the multi-modality of future motion states. Unlike existing stochastic trajectory prediction methods which usually use a latent variable to represent multi-modality, we explicitly simulate the process of human motion variation from indeterminate to determinate. In this paper, we present a new framework to formulate the trajectory prediction task as a reverse process of motion indeterminacy diffusion (MID), in which we progressively discard indeterminacy from all the walkable area…
Mojito: Motion Trajectory and Intensity Control for Video Generation
University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Microsoft（微软公司）；University of Washington（华盛顿大学）；Georgia Institute of Technology（佐治亚理工学院）；Northeastern University（东北大学）
