To defend against manipulation of image content, such as splicing, copy-move, and removal, we develop a Progressive Spatio-Channel Correlation Network (PSCC-Net) to detect and localize image manipulations. PSCC-Net processes the image in a two-path procedure: a top-down path that extracts local and global features and a bottom-up path that detects whether the input image is manipulated, and estimates its manipulation masks at multiple scales, where each mask is conditioned on the previous one. Different from the conventional encoder-decoder and no-pooling structures, PSCC-Net leverages features at different scales with dense cross-connections to produce manipulation masks in a coarse-to-fine fashion. Moreover, a Spatio-Channel Correlation Module (SCCM) captures both spatial and channel-wise correlations in the bottom-up path, which endows features with holistic cues, enabling the network to cope with a wide range of manipulation attacks. Thanks to the light-weight backbone and progressive mechanism, PSCC-Net can process 1,080P images at 50+ FPS. Extensive experiments demonstrate the superiority of PSCC-Net over the state-of-the-art methods on both detection and localization.
University of Maryland, College Park（马里兰大学帕克分校）；NVIDIA（英伟达）；University of California, Davis（加利福尼亚大学戴维斯分校）
In this paper, we propose Spatio-TEmporal Progressive (STEP) action detector---a progressive learning framework for spatio-temporal action detection in videos. Starting from a handful of coarse-scale proposal cuboids, our approach progressively refines the proposals towards actions over a few steps. In this way, high-quality proposals (i.e., adhere to action movements) can be gradually obtained at later steps by leveraging the regression outputs from previous steps. At each step, we adaptively extend the proposals in time to incorporate more related temporal context. Compared to the prior work…
University of Science and Technology of China（中国科学技术大学）；Huawei Cloud EI Product Department（华为云EI产品部）；Huawei Noah’s Ark Laboratory（华为诺亚方舟实验室）
Existing methods for person re-identification (Re-ID) are mostly based on supervised learning which requires numerous manually labeled samples across all camera views for training. Such a paradigm suffers the scalability issue since in real-world Re-ID application, it is difficult to exhaustively label abundant identities over multiple disjoint camera views. To this end, we propose a progressive deep learning method for unsupervised person Re-ID in the wild by Tracklet Association with Spatio-Temporal Regularization (TASTR). In our approach, we first collect tracklet data within each camera by…
Graph convolutional networks (GCNs) have been very successful in skeleton-based human action recognition where the sequence of skeletons is modeled as a graph. However, most of the GCN-based methods in this area train a deep feed-forward network with a fixed topology that leads to high computational complexity and restricts their application in low computation scenarios. In this paper, we propose a method to automatically find a compact and problem-specific topology for spatio-temporal graph convolutional networks in a progressive manner. Experimental results on two widely used datasets for sk…
Spatio-Temporal Progressive Attention Model for EEG Classification in Rapid Serial Visual Presentation Task
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）
University of Science and Technology Beijing（北京科技大学）；University of Sydney（悉尼大学）；Beijing Engineering Research Center of Industrial Spectrum Imaging（北京工业光谱成像工程研究中心）；Baidu（百度）
Existing methods of multi-person video 3D human Pose and Shape Estimation (PSE) typically adopt a two-stage strategy, which first detects human instances in each frame and then performs single-person PSE with temporal model. However, the global spatio-temporal context among spatial instances can not be captured. In this paper, we propose a new end-to-end multi-person 3D Pose and Shape estimation framework with progressive Video Transformer, termed PSVT. In PSVT, a spatio-temporal encoder (STE) captures the global feature dependencies among spatial objects. Then, spatio-temporal pose decoder (S…
Deep neural networks have been widely used for feature learning in facial expression recognition systems. However, small datasets and large intra-class variability can lead to overfitting. In this paper, we propose a method which learns an optimized compact network topology for real-time facial expression recognition utilizing localized facial landmark features. Our method employs a spatio-temporal bilinear layer as backbone to capture the motion of facial landmarks during the execution of a facial expression effectively. Besides, it takes advantage of Monte Carlo Dropout to capture the model'…
Sun Yat-sen University（中山大学）；ByteDance Inc.（字节跳动公司）；Baidu Inc.（百度公司）
In this paper, we introduce a novel task, referred to as Weakly-Supervised Spatio-Temporal Anomaly Detection (WSSTAD) in surveillance video. Specifically, given an untrimmed video, WSSTAD aims to localize a spatio-temporal tube (i.e., a sequence of bounding boxes at consecutive times) that encloses the abnormal event, with only coarse video-level annotations as supervision during training. To address this challenging task, we propose a dual-branch network which takes as input the proposals with multi-granularities in both spatial-temporal domains. Each branch employs a relationship reasoning m…
Brainnetome Center, National Laboratory of Pattern Recognition (NLPR)Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所国家模式识别实验室（NLPR）脑网络组中心）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology (CEBSIT)（脑科学与智能技术卓越中心）
Group activities usually involve spatiotemporal dynamics among many interactive individuals, while only a few participants at several key frames essentially define the activity. Therefore, effectively modeling the group-relevant and suppressing the irrelevant actions (and interactions) are vital for group activity recognition. In this paper, we propose a novel method based on deep reinforcement learning to progressively refine the low-level features and high-level relations of group activities. Firstly, we construct a semantic relation graph (SRG) to explicitly model the relations among person…
ByteDance Inc.（字节跳动有限公司）；University of California, Davis（加利福尼亚大学戴维斯分校）
Video inpainting aims to fill spatio-temporal "corrupted" regions with plausible content. To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown content. Current methods achieve this goal through attention, flow-based warping, or 3D temporal convolution. However, flow-based warping can create artifacts when optical flow is not accurate, while temporal convolution may suffer from spatial misalignment. We propose 'Progressive Temporal Feature Alignment Network', which progressively enriches features extracted from the current …
Existing studies for automated melanoma diagnosis are based on single-time point images of lesions. However, melanocytic lesions de facto are progressively evolving and, moreover, benign lesions can progress into malignant melanoma. Ignoring cross-time morphological changes of lesions thus may lead to misdiagnosis in borderline cases. Based on the fact that dermatologists diagnose ambiguous skin lesions by evaluating the dermoscopic changes over time via follow-up examination, in this study, we propose an automated framework for melanoma diagnosis using sequential dermoscopic images. To captur…
Nanjing University of Information Science and Technology（南京信息工程大学）；JD Finance America Corporation（京东金融美国公司）
Due to a variety of motions across different frames, it is highly challenging to learn an effective spatiotemporal representation for accurate video saliency prediction (VSP). To address this issue, we develop an effective spatiotemporal feature alignment network tailored to VSP, mainly including two key sub-networks: a multi-scale deformable convolutional alignment network (MDAN) and a bidirectional convolutional Long Short-Term Memory (Bi-ConvLSTM) network. The MDAN learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner, which can well handle v…
Tianjin University（天津大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Shadow removal in a single image has received increasing attention in recent years. However, removing shadows over dynamic scenes remains largely under-explored. In this paper, we propose the first data-driven video shadow removal model, termed PSTNet, by exploiting three essential characteristics of video shadows, i.e., physical property, spatio relation, and temporal coherence. Specifically, a dedicated physical branch was established to conduct local illumination estimation, which is more applicable for scenes with complex lighting and textures, and then enhance the physical features via a …
MPFNet: A Multi-Prior Fusion Network with a Progressive Training Strategy for Micro-Expression Recognition
Defense Innovation Institute, Academy of Military Sciences (AMS) and Intelligent Game and Decision Laboratory（国防科技创新院、军事科学研究院（AMS）和智能游戏与决策实验室）；School of Computer Science and Technology, Dalian University of Technology（计算机科学与技术学院、大连理工大学）
CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding
Department of Medical Device Management and Research, Samsung Advanced Institute for Health Sciences & Technology (SAIHST), Sungkyunkwan University（医学院设备管理与研究系，三星健康科学与技术研究院（SAIHST），全北大学）；Clinical Robotics and Embodied AI Research Center, Smart Healthcare Research Institute, Research Institute for Future Medicine, Samsung Medical Center（临床机器人与具身人工智能研究中心，智能医疗研究机构，未来医学研究所，三星医疗中心）；Department of Surgery, Samsung Medical Center（外科系，三星医疗中心）
Institute of Biomedical Engineering, University of Oxford（牛津大学生物医学工程研究所）
Fetal heart motion is an important diagnostic indicator for structural detection and functional assessment of congenital heart disease. We propose an approach towards integrating deep convolutional and recurrent architectures that utilize localized spatial and temporal features of different anatomical substructures within a global spatiotemporal context for interpretation of fetal echocardiography videos. We formulate our task as a cardiac structure localization problem with convolutional architectures for aggregating global spatial context and detecting anatomical structures on spatial region…
University of Sannio（萨尼奥大学）；Warsaw University of Technology（华沙理工大学）；La Sapienza University（罗马第一大学）；European Space Agency（欧洲空间局）
Cloud removal is a relevant topic in Remote Sensing as it fosters the usability of high-resolution optical images for Earth monitoring and study. Related techniques have been analyzed for years with a progressively clearer view of the appropriate methods to adopt, from multi-spectral to inpainting methods. Recent applications of deep generative models and sequence-to-sequence-based models have proved their capability to advance the field significantly. Nevertheless, there are still some gaps, mostly related to the amount of cloud coverage, the density and thickness of clouds, and the occurred …
Multi-scale Spatio-temporal Transformer-based Imbalanced Longitudinal Learning for Glaucoma Forecasting from Irregular Time Series Images
Zhejiang Lab（之江实验室）；The Chinese University of Hong Kong（香港中文大学）；Beijing Institute of Ophthalmology（北京眼科研究所）；Beijing Tongren Eye Center（北京同仁眼科中心）；Beijing Tongren Hospital（北京同仁医院）；Capital Medical University（首都医科大学）；Beijing Key Laboratory of Ophthalmology and Visual Sciences（北京市眼科与视觉科学重点实验室）
Resi-VidTok: An Efficient and Decomposed Progressive Tokenization Framework for Ultra-Low-Rate and Lightweight Video Transmission
GIC & 6GIC, Institute for Communication Systems (ICS), University of Surrey（5GIC与6GIC，通信系统研究所（ICS），萨里大学）；Department of Electrical and Computer Engineering, University of California at Davis（电气与计算机工程系，加州大学戴维斯分校）
Beihang University（北京航空航天大学）；State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）；International Research Institute for Multidisciplinary Science（国际交叉科学研究院）
Image-based salient object detection (SOD) has been extensively studied in the past decades. However, video-based SOD is much less explored since there lack large-scale video datasets within which salient objects are unambiguously defined and annotated. Toward this end, this paper proposes a video-based SOD dataset that consists of 200 videos (64 minutes). In constructing the dataset, we manually annotate all objects and regions over 7,650 uniformly sampled keyframes and collect the eye-tracking data of 23 subjects that free-view all videos. From the user data, we find salient objects in video…
Depth can provide useful geographical cues for salient object detection (SOD), and has been proven helpful in recent RGB-D SOD methods. However, existing video salient object detection (VSOD) methods only utilize spatiotemporal information and seldom exploit depth information for detection. In this paper, we propose a depth-cooperated trimodal network, called DCTNet for VSOD, which is a pioneering work to incorporate depth information to assist VSOD. To this end, we first generate depth from RGB frames, and then propose an approach to treat the three modalities unequally. Specifically, a multi…
Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
School of Computer Science, China University of Geosciences（中国地质大学（武汉）计算机学院）；Engineering Research Center of Natural Resource Information Management and Digital Twin Engineering Software, Ministry of Education（教育部自然资源信息管理与数字孪生工程软件工程研究中心）；Second Surveying and Mapping Institute of Hunan Province（湖南省第二测绘院）；Key Laboratory of Natural Resources Monitoring and Supervision in Southern Hilly Region, Ministry of Natural Resources（自然资源部南方山地自然资源监测与监管重点实验室）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Chinese University of Hong Kong（香港中文大学）；Horizon Robotics（地平线机器人）
Transferring existing image-based detectors to the video is non-trivial since the quality of frames is always deteriorated by part occlusion, rare pose, and motion blur. Previous approaches exploit to propagate and aggregate features across video frames by using optical flow-warping. However, directly applying image-level optical flow onto the high-level features might not establish accurate spatial correspondences. Therefore, a novel module called Learnable Spatio-Temporal Sampling (LSTS) has been proposed to learn semantic-level correspondences among adjacent frame features accurately. The s…
