Many video understanding tasks work in the offline setting by assuming that the input video is given from the start to the end. However, many real-world problems require the online setting, making a decision immediately using only the current and the past frames of videos such as in autonomous driving and surveillance systems. In this paper, we present a novel solution for online action detection by using a simple yet effective RNN-based networks called the Future Anticipation and Temporally Smoothing network (FATSnet). The proposed network consists of a module for anticipating the future that can be trained in an unsupervised manner with the cycle-consistency loss, and another component for aggregating the past and the future for temporally smooth frame-by-frame predictions. We also propose a solution to relieve the performance loss when running RNN-based models on very long sequences. Evaluations on TVSeries, THUMOS14, and BBDB show that our method achieve the state-of-the-art performances compared to the previous works on online action detection.
In this work, we address the problem of spatio-temporal action detection in temporally untrimmed videos. It is an important and challenging task as finding accurate human actions in both temporal and spatial space is important for analyzing large-scale video data. To tackle this problem, we propose a cascade proposal and location anticipation (CPLA) model for frame-level action detection. There are several salient points of our model: (1) a cascade region proposal network (casRPN) is adopted for action proposal generation and shows better localization accuracy compared with single region propo…
University of Bonn（波恩大学）；The University of Melbourne（墨尔本大学）；MPI Informatics（马克斯·普朗克信息学研究所）
With the success of deep learning methods in analyzing activities in videos, more attention has recently been focused towards anticipating future activities. However, most of the work on anticipation either analyzes a partially observed activity or predicts the next action class. Recently, new approaches have been proposed to extend the prediction horizon up to several minutes in the future and that anticipate a sequence of future activities including their durations. While these works decouple the semantic interpretation of the observed sequence from the anticipation task, we propose a framew…
University of California, Riverside（加州大学河滨分校）；JD.com AI Research（京东人工智能研究院）
While action anticipation has garnered a lot of research interest recently, most of the works focus on anticipating future action directly through observed visual cues only. In this work, we take a step back to analyze how the human capability to anticipate the future can be transferred to machine learning algorithms. To incorporate this ability in intelligent systems a question worth pondering upon is how exactly do we anticipate? Is it by anticipating future actions from past experiences? Or is it by simulating possible scenarios based on cues from the present? A recent study on human psycho…
Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions
Center for Humanoid Research, Korea Institute of Science and Technology（韩国科学技术研究院仿人机器人研究中心）；School of Electrical and Electronic Engineering, Yonsei University（延世大学电气与电子工程学院）
Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?
Universidad de Alicante（阿利坎特大学）；Foundation for Research and Technology-Hellas（希腊基础研究与技术基金会）；University of Crete（克里特大学）；Hellenic Mediterranean University（希腊地中海大学）
EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation
Department of Mathematics and Computer Science, University of Catania（卡塔尼亚大学数学与计算机科学系）；Next Vision s.r.l.（Next Vision公司）；Reality Labs Research, Meta（Meta现实实验室）
SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention
UCL Hawkes Institute（UCL哈维斯研究所）；University College London（伦敦大学学院）；Dept of Medical Physics & Biomedical Engineering（医学物理与生物医学工程系）；UCL（伦敦大学学院）；Dept of Computer Science（计算机科学系）；National Hospital for Neurology and Neurosurgery（神经病学与神经外科国家医院）；Division of Informatics, Imaging and Data Science（信息学、成像与数据科学 division）
Dynamic reconstructions (3D+T) of coronary arteries could give important perfusion details to clinicians. Temporal matching of the different views, which may not be acquired simultaneously, is a prerequisite for an accurate stereo-matching of the coronary segments. In this paper, we show how a neural network can be trained from angiographic sequences to synchronize different views during the cardiac cycle using raw x-ray angiography videos exclusively. First, we train a neural network model with angiographic sequences to extract features describing the progression of the cardiac cycle. Then, w…
Human pose detection systems based on state-of-the-art DNNs are on the go to be extended, adapted and re-trained to fit the application domain of specific sports. Therefore, plenty of noisy pose data will soon be available from videos recorded at a regular and frequent basis. This work is among the first to develop mining algorithms that can mine the expected abundance of noisy and annotation-free pose data from video recordings in individual sports. Using swimming as an example of a sport with dominant cyclic motion, we show how to determine unsupervised time-continuous cycle speeds and tempo…
University of Florence（佛罗伦萨大学）；IMT Lille Douai（里尔杜埃高等矿业电信学院）；Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）
In this paper, a model is presented to extract statistical summaries to characterize the repetition of a cyclic body action, for instance a gym exercise, for the purpose of checking the compliance of the observed action to a template one and highlighting the parts of the action that are not correctly executed (if any). The proposed system relies on a Riemannian metric to compute the distance between two poses in such a way that the geometry of the manifold where the pose descriptors lie is preserved; a model to detect the begin and end of each cycle; a model to temporally align the poses of di…
Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community. We propose a self-supervised solution to this problem using temporal cycle consistency jointly in vision and language, training on narrated video. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. This constraint leads to the discovery of high-level transitions between moments in time, since such transitions are easily inverted and shared across modalities. We justify the design of our model wit…
Indian Institute of Technology (BHU)（印度理工学院（贝纳拉斯印度教大学））；Nanyang Technological University（南洋理工大学）
Recent advancements in computational resources and Deep Learning methodologies has significantly benefited development of intelligent vision-based surveillance applications. Gait recognition in the presence of occlusion is one of the challenging research topics in this area, and the solutions proposed by researchers to date lack in robustness and also dependent of several unrealistic constraints, which limits their practical applicability. We improve the state-of-the-art by developing novel deep learning-based algorithms to identify the occluded frames in an input sequence and next reconstruct…
Tsinghua University（清华大学）；Australian National University（澳大利亚国立大学）；Shanghai AI Lab（上海人工智能实验室）；Baidu Research（百度研究院）
In this paper, we are interested in learning a generalizable person re-identification (re-ID) representation from unlabeled videos. Compared with 1) the popular unsupervised re-ID setting where the training and test sets are typically under the same domain, and 2) the popular domain generalization (DG) re-ID setting where the training samples are labeled, our novel scenario combines their key challenges: the training samples are unlabeled, and collected form various domains which do no align with the test domain. In other words, we aim to learn a representation in an unsupervised manner and di…
Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as $L^2$ over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail…
Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Southern California（南加州大学）；Tsinghua University（清华大学）；Beihang University（北航）；Wuhan University（武汉大学）；Shanghai Jiao Tong University（上海交通大学）；European Institute of Innovation and Technology Ningbo（创新与技术欧洲研究所宁波）；FNii, The Chinese University of Hong Kong, Shenzhen（FNii，香港中文大学（深圳））；National University of Singapore（新加坡国立大学）
Virginia Tech（弗吉尼亚理工大学）；NEC Labs America（NEC美国实验室）；UCSD（加州大学圣迭戈分校）
Monocular visual odometry (VO) suffers severely from error accumulation during frame-to-frame pose estimation. In this paper, we present a self-supervised learning method for VO with special consideration for consistency over longer sequences. To this end, we model the long-term dependency in pose prediction using a pose network that features a two-layer convolutional LSTM module. We train the networks with purely self-supervised losses, including a cycle consistency loss that mimics the loop closure module in geometric VO. Inspired by prior geometric systems, we allow the networks to see beyo…
Deformable Image Registration for Self-supervised Cardiac Phase Detection in Multi-View Multi-Disease Cardiac Magnetic Resonance Images
Department of Internal Medicine III, Heidelberg University Hospital（海德堡大学第三内科部门医院）；Medical Faculty of Heidelberg University（海德堡大学医学院）；DZHK (German Centre for Cardiovascular Research)（德国心血管研究中心）；Division of Pediatric Cardiology, Department of Pediatrics, UT Southwestern /Children’s Health（儿童心脏病科，德克萨斯西南医学中心/儿童健康）；Department of Cardiothoracic, Transplantation and Vascular Surgery, Hannover Medical School（汉诺威医学院心胸外科、移植外科和血管外科部门）
本文提出一种自监督深度学习方法，通过多视角多疾病心脏磁共振图像检测五个关键帧，提高了心脏相位检测的准确率，适用于SAX和4CH cine CMR。
MCM: Mamba-based Cardiac Motion Tracking using Sequential Images in MRI
School of Computer Science, University of Birmingham（英国伯明翰大学计算机科学学院）；Division of Informatics, Imaging and Data Sciences, University of Manchester（曼彻斯特大学信息学与数据科学系）；Centre for Computational Imaging and Modelling in Medicine, University of Manchester（曼彻斯特大学医学计算成像与建模中心）；Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（广东省计算机视觉与虚拟现实技术重点实验室，深圳先进技术研究院，中国科学院）；MRC Laboratory of Medical Sciences, Imperial College London（伦敦帝国理工学院医学科学实验室）；UKRI CDT in AI for Healthcare, Imperial College London（伦敦帝国理工学院人工智能与医疗健康CDT）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机科学系）
