We wish to detect specific categories of objects, for online vision systems that will run in the real world. Object detection is already very challenging. It is even harder when the images are blurred, from the camera being in a car or a hand-held phone. Most existing efforts either focused on sharp images, with easy to label ground truth, or they have treated motion blur as one of many generic corruptions.
Instead, we focus especially on the details of egomotion induced blur. We explore five classes of remedies, where each targets different potential causes for the performance gap between sharp and blurred images. For example, first deblurring an image changes its human interpretability, but at present, only partly improves object detection. The other four classes of remedies address multi-scale texture, out-of-distribution testing, label generation, and conditioning by blur-type. Surprisingly, we discover that custom label generation aimed at resolving spatial ambiguity, ahead of all others, markedly improves object detection. Also, in contrast to findings from classification, we see a noteworthy boost by conditioning our model on bespoke categories of motion blur.
We validate and cross-breed the different remedies experimentally on blurred COCO images and real-world blur datasets, producing an easy and practical favorite model with superior detection rates.
Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）
Currently, many blind deblurring methods assume blurred images are noise-free and perform unsatisfactorily on the blurry images with noise. Unfortunately, noise is quite common in real scenes. A straightforward solution is to denoise images before deblurring them. However, even state-of-the-art denoisers cannot guarantee to remove noise entirely. Slight residual noise in the denoised images could cause significant artifacts in the deblurring stage. To tackle this problem, we propose a cascaded framework consisting of a denoiser subnetwork and a deblurring subnetwork. In contrast to previous me…
In Video Instance Segmentation (VIS), current approaches either focus on the quality of the results, by taking the whole video as input and processing it offline; or on speed, by handling it frame by frame at the cost of competitive performance. In this work, we propose an online method that is on par with the performance of the offline counterparts. We introduce a message-passing graph neural network that encodes objects and relates them through time. We additionally propose a novel module to fuse features from the feature pyramid network with residual connections. Our model, trained end-to-e…
O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation
Academy for Engineering & Technology（工程与技术学院）；Fudan University（复旦大学）；State Key Laboratory of Intelligent Vehicle Safety Technology（智能车辆安全技术国家重点实验室）；School of Future Technology（未来技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）
Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like UVO and OVIS. We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO an…
Abrupt motion of camera or objects in a scene result in a blurry video, and therefore recovering high quality video requires two types of enhancements: visual enhancement and temporal upsampling. A broad range of research attempted to recover clean frames from blurred image sequences or temporally upsample frames by interpolation, yet there are very limited studies handling both problems jointly. In this work, we present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. We design our framework by first learning t…
Beijing University of Technology（北京工业大学）；Tsinghua University（清华大学）
Blind image deconvolution is the problem of recovering the latent image from the only observed blurry image when the blur kernel is unknown. In this paper, we propose an edge-based blur kernel estimation method for blind motion deconvolution. In our previous work, we incorporate both sparse representation and self-similarity of image patches as priors into our blind deconvolution model to regularize the recovery of the latent image. Since almost any natural image has properties of sparsity and multi-scale self-similarity, we construct a sparsity regularizer and a cross-scale non-local regulari…
EndoCaver: Handling Fog, Blur and Glare in Endoscopic Images via Joint Deblurring-Segmentation
1 CyPhi ( ) AI Research Lab, School of IT, Monash University, Malaysia Campus；2 Department of Electronic \& Computer Engineering, The Hong Kong University of Science；3 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Present-day deep learning-based motion deblurring methods utilize the pair of synthetic blur and sharp data to regress any particular framework. This task is designed for directly translating a blurry image input into its restored version as output. The aforementioned approach relies heavily on the quality of the synthetic blurry data, which are only available before the training stage. Handling this issue by providing a large amount of data is expensive for common usage. We answer this challenge by providing an on-the-fly blurry data augmenter that can be run during training and test stages. …
School of Electrical and Computer Engineering（电气与计算机工程学院）；University of Tehran（德黑兰大学）
Multiple object tracking is a challenging problem in computer vision due to difficulty in dealing with motion prediction, occlusion handling, and object re-identification. Many recent algorithms use motion and appearance cues to overcome these challenges. But using appearance cues increases the computation cost notably and therefore the speed of the algorithm decreases significantly which makes them inappropriate for online applications. In contrast, there are algorithms that only use motion cues to increase speed, especially for online applications. But these algorithms cannot handle occlusio…
Kinematics-Driven Gaussian Shape Deformation for Blurry Monocular Dynamic Scenes
AI Institute, Seoul National University（首尔国立大学人工智能研究所）；Interdisciplinary Program in Neuroscience, Seoul National University（首尔国立大学神经科学跨学科项目）；Gwangju Institute of Science（全州科学技术学院）；NAVER AI Lab（NAVER AI实验室）
NAVER CLOVA；NAVER AI Lab（NAVER人工智能实验室）；GIST（光州科学技术院）；Seoul National University（首尔国立大学）；Yonsei University（延世大学）
Learning under a continuously changing data distribution with incorrect labels is a desirable real-world problem yet challenging. A large body of continual learning (CL) methods, however, assumes data streams with clean labels, and online learning scenarios under noisy data streams are yet underexplored. We consider a more practical CL task setup of an online learning from blurry data stream with corrupted labels, where existing CL methods struggle. To address the task, we first argue the importance of both diversity and purity of examples in the episodic memory of continual learning models. T…
Beijing Normal University（北京师范大学）；Tianjin University（天津大学）
Most blind deconvolution methods usually pre-define a large kernel size to guarantee the support domain. Blur kernel estimation error is likely to be introduced, yielding severe artifacts in deblurring results. In this paper, we first theoretically and experimentally analyze the mechanism to estimation error in oversized kernel, and show that it holds even on blurry images without noises. Then to suppress this adverse effect, we propose a low rank-based regularization on blur kernel to exploit the structural information in degraded kernels, by which larger-kernel effect can be effectively supp…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Seoul National University（首尔国立大学）
State-of-the-art video deblurring methods are capable of removing non-uniform blur caused by unwanted camera shake and/or object motion in dynamic scenes. However, most existing methods are based on batch processing and thus need access to all recorded frames, rendering them computationally demanding and time consuming and thus limiting their practical use. In contrast, we propose an online (sequential) video deblurring method based on a spatio-temporal recurrent network that allows for real-time performance. In particular, we introduce a novel architecture which extends the receptive field wh…
University of Science and Technology of China（中国科学技术大学）；KDDI research, Inc.（KDDI研究公司）
Various blur distortions in video will cause negative impact on both human viewing and video-based applications, which makes motion-robust deblurring methods urgently needed. Most existing works have strong dataset dependency and limited generalization ability in handling challenging scenarios, like blur in low contrast or severe motion areas, and non-uniform blur. Therefore, we propose a PRiOr-enlightened and MOTION-robust video deblurring model (PROMOTION) suitable for challenging blurs. On the one hand, we use 3D group convolution to efficiently encode heterogeneous prior information, expli…
Face recognition system is one of the esteemed research areas in pattern recognition and computer vision as long as its major challenges. A few challenges in recognizing faces are blur, illumination, and varied expressions. Blur is natural while taking photographs using cameras, mobile phones, etc. Blur can be uniform and non-uniform. Usually non-uniform blur happens in images taken using handheld image devices. Distinguishing or handling a blurred image in a face recognition system is generally tough. Under varying lighting conditions, it is challenging to identify the person correctly. Diver…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Blind super-resolution (SR) methods aim to generate a high quality high resolution image from a low resolution image containing unknown degradations. However, natural images contain various types and amounts of blur: some may be due to the inherent degradation characteristics of the camera, but some may even be intentional, for aesthetic purposes (e.g. Bokeh effect). In the case of the latter, it becomes highly difficult for SR methods to disentangle the blur to remove, and that to leave as is. In this paper, we propose a novel blind SR framework based on kernel-oriented adaptive local adjustm…
Motion-Aware Adaptive Pixel Pruning for Efficient Local Motion Deblurring
Harbin Institute of Technology \& City University of Hong Kong；School of Computer Science；Technology, Harbin Institute of Technology；Tianjin University \& Low-Altitude Intelligence Lab, Xiong'an National Innovation Center Technology Co., Ltd；College of Intelligence；Computing, Tianjin University
Tianjin University（天津大学）；Harbin Institute of Technology（哈尔滨工业大学）
Blind deconvolution is a classical yet challenging low-level vision problem with many real-world applications. Traditional maximum a posterior (MAP) based methods rely heavily on fixed and handcrafted priors that certainly are insufficient in characterizing clean images and blur kernels, and usually adopt specially designed alternating minimization to avoid trivial solution. In contrast, existing deep motion deblurring networks learn from massive training images the mapping to clean image or blur kernel, but are limited in handling various complex and large size blur kernels. To connect MAP an…
Xiamen University（厦门大学）；University of Science and Technology of China（中国科学技术大学）；Columbia University（哥伦比亚大学）
We propose a new framework called Noise2Blur (N2B) for training robust image denoising models without pre-collected paired noisy/clean images. The training of the model requires only some (or even one) noisy images, some random unpaired clean images, and noise-free but blurred labels obtained by predefined filtering of the noisy images. The N2B model consists of two parts: a denoising network and a noise extraction network. First, the noise extraction network learns to output a noise map using the noise information from the denoising network under the guidence of the blurred labels. Then, the …
Learning Motion Blur Robust Vision Transformers for Real-Time UAV Tracking
College of Computer Science（计算机科学学院）；Guilin University of Technology（桂林理工大学）；School of Computer Science（计算机科学学院）；Fudan University（复旦大学）；Sun Yat-sen University（中山大学）；Sichuan University（四川大学）
Blind image deblurring is a particularly challenging inverse problem where the blur kernel is unknown and must be estimated en route to recover the deblurred image. The problem is of strong practical relevance since many imaging devices such as cellphone cameras, must rely on deblurring algorithms to yield satisfactory image quality. Despite significant research effort, handling large motions remains an open problem. In this paper, we develop a new method called Blind Image Deblurring using Row-Column Sparsity (BD-RCS) to address this issue. Specifically, we model the outer product of kernel a…
DeepRFTv2: Kernel-level Learning for Image Deblurring
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University（上海多维信息处理重点实验室，华东师范大学）；State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, Chinese Academy of Sciences（红外物理国家重点实验室，上海技术物理研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
This paper proposes an online visual multi-object tracking algorithm using a top-down Bayesian formulation that seamlessly integrates state estimation, track management, clutter rejection, occlusion and mis-detection handling into a single recursion. This is achieved by modeling the multi-object state as labeled random finite set and using the Bayes recursion to propagate the multi-object filtering density forward in time. The proposed filter updates tracks with detections but switches to image data when mis-detection occurs, thereby exploiting the efficiency of detection data and the accuracy…
Hybrid CNN-ViT Framework for Motion-Blurred Scene Text Restoration
Department of Electrical Engineering, University of Engineering & Technology, New Campus, Lahore, Pakistan（电子工程系，工程科技大学，新校区，拉合尔，巴基斯坦）；Department of Electrical, Electronic, and Future Technologies, Sheffield Hallam University, Sheffield S1 1WB, UK（电子、电气与未来技术系，谢菲尔德哈姆大学，谢菲尔德 S1 1WB，英国）
Heritage Institute of Technology, Kolkata（加尔各答传统理工学院）；Indian Statistical Institute, Kolkata（印度统计研究所加尔各答分所）
Automatic detection of scene texts in the wild is a challenging problem, particularly due to the difficulties in handling (i) occlusions of varying percentages, (ii) widely different scales and orientations, (iii) severe degradations in the image quality etc. In this article, we propose a fully convolutional neural network architecture consisting of a novel Feature Representation Block (FRB) capable of efficient abstraction of information. The proposed network has been trained using curriculum learning with respect to difficulties in image samples and gradual pixel-wise blurring. It is capable…
We propose a novel end-to-end learning-based approach for single image defocus deblurring. The proposed approach is equipped with a novel Iterative Filter Adaptive Network (IFAN) that is specifically designed to handle spatially-varying and large defocus blur. For adaptively handling spatially-varying blur, IFAN predicts pixel-wise deblurring filters, which are applied to defocused features of an input image to generate deblurred features. For effectively managing large blur, IFAN models deblurring filters as stacks of small-sized separable filters. Predicted separable deblurring filters are a…
Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the…
In this paper, we propose an online multi-object tracking (MOT) method in a delta Generalized Labeled Multi-Bernoulli (delta-GLMB) filter framework to address occlusion and miss-detection issues, reduce false alarms, and recover identity switch (ID switch). To handle occlusion and miss-detection issues, we propose a measurement-to-disappeared track association method based on one-step delta-GLMB filter, so it is possible to manage these difficulties by jointly processing occluded or miss-detected objects. This part of proposed method is based on a proposed similarity metric which is responsibl…
Tianjin University（天津大学）；Nanyang Technological University（南洋理工大学）；Kyushu University（九州大学）；Alibaba Group（阿里巴巴集团）；University of Alberta（阿尔伯塔大学）；Zhejiang Sci-Tech University（浙江理工大学）
Motion blur caused by the moving of the object or camera during the exposure can be a key challenge for visual object tracking, affecting tracking accuracy significantly. In this work, we explore the robustness of visual object trackers against motion blur from a new angle, i.e., adversarial blur attack (ABA). Our main objective is to online transfer input frames to their natural motion-blurred counterparts while misleading the state-of-the-art trackers during the tracking process. To this end, we first design the motion blur synthesizing method for visual tracking based on the generation prin…
The University of Sydney（悉尼大学）；University of Technology Sydney（悉尼科技大学）
Online proctoring has become a necessity in online teaching. Video-based crowd-sourced online proctoring solutions are being used, where an exam-taking student's video is monitored by third parties, leading to privacy concerns. In this paper, we propose a privacy-preserving online proctoring system. The proposed image-hashing-based system can detect the student's excessive face and body movement (i.e., anomalies) that is resulted when the student tries to cheat in the exam. The detection can be done even if the student's face is blurred or masked in video frames. Experiment with an in-house da…
Global optimization algorithms have shown impressive performance in data-association based multi-object tracking, but handling online data remains a difficult hurdle to overcome. In this paper, we present a hybrid data association framework with a min-cost multi-commodity network flow for robust online multi-object tracking. We build local target-specific models interleaved with global optimization of the optimal data association over multiple video frames. More specifically, in the min-cost multi-commodity network flow, the target-specific similarities are online learned to enforce the local …
University of Science and Technology of China（中国科学技术大学）；Noah’s Ark Lab, Huawei Technologies（华为技术有限公司诺亚方舟实验室）；Huawei Cloud BU（华为云BU）
Moire artifacts are common in digital photography, resulting from the interference between high-frequency scene content and the color filter array of the camera. Existing deep learning-based demoireing methods trained on large scale datasets are limited in handling various complex moire patterns, and mainly focus on demoireing of photos taken of digital displays. Moreover, obtaining moire-free ground-truth in natural scenes is difficult but needed for training. In this paper, we propose a self-adaptive learning method for demoireing a high-frequency image, with the help of an additional defocu…
University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Baidu Inc.（百度公司）；National Laboratory of Pattern Recognition (NLPR)（模式识别国家重点实验室）；Chinese Academy of Sciences Artificial Intelligence Research (CAS-AIR)（中国科学院人工智能研究院）
Video text detection is considered as one of the most difficult tasks in document analysis due to the following two challenges: 1) the difficulties caused by video scenes, i.e., motion blur, illumination changes, and occlusion; 2) the properties of text including variants of fonts, languages, orientations, and shapes. Most existing methods attempt to enhance the performance of video text detection by cooperating with video text tracking, but treat these two tasks separately. In this work, we propose an end-to-end video text detection model with online tracking to address these two challenges. …
This paper tackles the problem of dynamic scene deblurring. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by a simple increment in the number of generic convolution layers, kernel-size, which comes with the burden of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations within and across differe…
This paper proposes an online multi-camera multi-object tracker that only requires monocular detector training, independent of the multi-camera configurations, allowing seamless extension/deletion of cameras without retraining effort. The proposed algorithm has a linear complexity in the total number of detections across the cameras, and hence scales gracefully with the number of cameras. It operates in the 3D world frame, and provides 3D trajectory estimates of the objects. The key innovation is a high fidelity yet tractable 3D occlusion model, amenable to optimal Bayesian multi-view multi-ob…
Bilkent University（比尔肯特大学）；Texas A&M University at Qatar（德州农工大学卡塔尔分校）
In this paper, an Entropy functional based online Adaptive Decision Fusion (EADF) framework is developed for image analysis and computer vision applications. In this framework, it is assumed that the compound algorithm consists of several sub-algorithms each of which yielding its own decision as a real number centered around zero, representing the confidence level of that particular sub-algorithm. Decision values are linearly combined with weights which are updated online according to an active fusion method based on performing entropic projections onto convex sets describing sub-algorithms. I…
Tracking of objects in 3D is a fundamental task in computer vision that finds use in a wide range of applications such as autonomous driving, robotics or augmented reality. Most recent approaches for 3D multi object tracking (MOT) from LIDAR use object dynamics together with a set of handcrafted features to match detections of objects. However, manually designing such features and heuristics is cumbersome and often leads to suboptimal performance. In this work, we instead strive towards a unified and learning based approach to the 3D MOT problem. We design a graph structure to jointly process …
