Both accuracy and efficiency are significant for pose estimation and tracking in videos. State-of-the-art performance is dominated by two-stages top-down methods. Despite the leading results, these methods are impractical for real-world applications due to their separated architectures and complicated calculation. This paper addresses the task of articulated multi-person pose estimation and tracking towards real-time speed. An end-to-end multi-task network (MTN) is designed to perform human detection, pose estimation, and person re-identification (Re-ID) tasks simultaneously. To alleviate the performance bottleneck caused by scale variation problem, a paradigm which exploits scale-normalized image and feature pyramids (SIFP) is proposed to boost both performance and speed. Given the results of MTN, we adopt an occlusion-aware Re-ID feature strategy in the pose tracking module, where pose information is utilized to infer the occlusion state to make better use of Re-ID feature. In experiments, we demonstrate that the pose estimation and tracking performance improves steadily utilizing SIFP through different backbones. Using ResNet-18 and ResNet-50 as backbones, the overall pose tracking framework achieves competitive performance with 29.4 FPS and 12.2 FPS, respectively. Additionally, occlusion-aware Re-ID feature decreases the identification switches by 37% in the pose tracking process.
Facial expressions are widely used in the behavioral interpretation of emotions, cognitive science, and social interactions. In this paper, we present a novel method for fully automatic facial expression recognition in facial image sequences. As the facial expression evolves over time facial landmarks are automatically tracked in consecutive video frames, using displacements based on elastic bunch graph matching displacement estimation. Feature vectors from individual landmarks, as well as pairs of landmarks tracking results are extracted, and normalized, with respect to the first frame in the…
Target tracking in hyperspectral videos is a new research topic. In this paper, a novel method based on convolutional network and Kernelized Correlation Filter (KCF) framework is presented for tracking objects of interest in hyperspectral videos. We extract a set of normalized three-dimensional cubes from the target region as fixed convolution filters which contain spectral information surrounding a target. The feature maps generated by convolutional operations are combined to form a three-dimensional representation of an object, thereby providing effective encoding of local spectral-spatial i…
We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments. To this end, we detect and reconstruct objects in each frame while predicting dense correspondences mappings into a normalized object space. We leverage those correspondences to inform a graph neural network to solve for the optimal, temporally-consistent 7-DoF pose trajectories of all objects. The novelty of our method is two-fold: first, we propose a new graph-based approach for differentiable pose estimation over time to learn optimal pose trajectories; second, we p…
School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院）
Deep convolutional neural networks (CNNs) have dominated many computer vision domains because of their great power to extract good features automatically. However, many deep CNNs-based computer vison tasks suffer from lack of training data while there are millions of parameters in the deep models. Obviously, these two biphase violation facts will result in parameter redundancy of many poorly designed deep CNNs. Therefore, we look deep into the existing CNNs and find that the redundancy of network parameters comes from the correlation between features in different channels within a convolutiona…
Nanjing University of Information Science and Technology（南京信息工程大学）；University of California, Merced（加州大学默塞德分校）
Deep networks have been successfully applied to visual tracking by learning a generic representation offline from numerous training images. However the offline training is time-consuming and the learned generic representation may be less discriminative for tracking specific objects. In this paper we present that, even without offline training with a large amount of auxiliary data, simple two-layer convolutional networks can be powerful enough to develop a robust representation for visual tracking. In the first frame, we employ the k-means algorithm to extract a set of normalized patches from t…
Microsoft Research Asia（微软亚洲研究院）；Huazhong University of Science and Technology（华中科技大学）；Eastern Institute for Advanced Study（东方理工高等研究院）
Multi-object tracking in videos requires to solve a fundamental problem of one-to-one assignment between objects in adjacent frames. Most methods address the problem by first discarding impossible pairs whose feature distances are larger than a threshold, followed by linking objects using Hungarian algorithm to minimize the overall distance. However, we find that the distribution of the distances computed from Re-ID features may vary significantly for different videos. So there isn't a single optimal threshold which allows us to safely discard impossible pairs. To address the problem, we prese…
Istituto Sistemi Complessi, Consiglio Nazionale delle Ricerche（意大利国家研究委员会复杂系统研究所）
The interest in 3D dynamical tracking is growing in fields such as robotics, biology and fluid dynamics. Recently, a major source of progress in 3D tracking has been the study of collective behaviour in biological systems, where the trajectories of individual animals moving within large and dense groups need to be reconstructed to understand the behavioural interaction rules. Experimental data in this field are generally noisy and at low spatial resolution, so that individuals appear as small featureless objects and trajectories must be retrieved by making use of epipolar information only. Mor…
SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos
College of Computer Science and Engineering, Guilin University of Technology（桂林理工大学计算机科学与工程学院）；Guangxi Key Laboratory of Embedded Technology and Intelligent System, Guilin University of Technology（广西嵌入式技术与智能系统重点实验室）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；School of Electronic Information, Central South University（中南大学电子信息学院）
Max-Planck-Institute for Informatics（马克斯·普朗克信息学研究所）；Universidad Rey Juan Carlos（胡安卡洛斯国王大学）
We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints, common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusi…
Peking University（北京大学）；Stanford University（斯坦福大学）；Shandong University（山东大学）；UCSD（加州大学圣地亚哥分校）；Tencent AI Lab（腾讯AI实验室）；Beijing Institute for General AI（北京通用人工智能研究院）
In this work, we tackle the problem of category-level online pose tracking of objects from point cloud sequences. For the first time, we propose a unified framework that can handle 9DoF pose tracking for novel rigid object instances as well as per-part pose tracking for articulated objects from known categories. Here the 9DoF pose, comprising 6D pose and 3D size, is equivalent to a 3D amodal bounding box representation with free 6D pose. Given the depth point cloud at the current frame and the estimated pose from the last frame, our novel end-to-end pipeline learns to accurately update the pos…
University of North Dakota（北达科他大学）；American Institute of Aeronautics and Astronautics（美国航空航天学会）
This paper presents the development of a real time tracking algorithm that runs on a 1.2 GHz PC/104 computer on-board a small UAV. The algorithm uses zero mean normalized cross correlation to detect and locate an object in the image. A kalman filter is used to make the tracking algorithm computationally efficient. Object position in an image frame is predicted using the motion model and a search window, centered at the predicted position is generated. Object position is updated with the measurement from object detection. The detected position is sent to the motion controller to move the gimbal…
VastMAT: A Large-Scale Multi-Category Benchmark for Multi-Animal Tracking
Dalian University of Technology（大连理工大学）；Wuhan University（武汉大学）；University of North Texas（北德克萨斯大学）；The Hong Kong University of Science and Technology（香港科技大学）
OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback
Hunan University（湖南大学）；Zhejiang University（浙江大学）；Ant Group（蚂蚁集团）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；INSAIT, Sofia University ``St. Kliment Ohridski''（INSAIT，索菲亚大学『圣克莱门特·奥赫里迪斯』）
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
