Significant progress has been made for estimating optical flow using deep neural networks. Advanced deep models achieve accurate flow estimation often with a considerable computation complexity and time-consuming training processes. In this work, we present a lightweight yet effective model for real-time optical flow estimation, termed FDFlowNet (fast deep flownet). We achieve better or similar accuracy on the challenging KITTI and Sintel benchmarks while being about 2 times faster than PWC-Net. This is achieved by a carefully-designed structure and newly proposed components. We first introduce an U-shape network for constructing multi-scale feature which benefits upper levels with global receptive field compared with pyramid network. In each scale, a partial fully connected structure with dilated convolution is proposed for flow estimation that obtains a good balance among speed, accuracy and number of parameters compared with sequential connected and dense connected structures. Experiments demonstrate that our model achieves state-of-the-art performance while being fast and lightweight.
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）；The University of Adelaide（阿德莱德大学）
Dense optical flow estimation plays a key role in many robotic vision tasks. In the past few years, with the advent of deep learning, we have witnessed great progress in optical flow estimation. However, current networks often consist of a large number of parameters and require heavy computation costs, largely hindering its application on low power-consumption devices such as mobile phones. In this paper, we tackle this challenge and design a lightweight model for fast and accurate optical flow prediction. Our proposed FastFlowNet follows the widely-used coarse-to-fine paradigm with following …
Microsoft Research Asia（微软亚洲研究院）；Dalian University of Technology（大连理工大学）；Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）
Object tracking has achieved significant progress over the past few years. However, state-of-the-art trackers become increasingly heavy and expensive, which limits their deployments in resource-constrained applications. In this work, we present LightTrack, which uses neural architecture search (NAS) to design more lightweight and efficient object trackers. Comprehensive experiments show that our LightTrack is effective. It can find trackers that achieve superior performance compared to handcrafted SOTA trackers, such as SiamRPN++ and Ocean, while using much fewer model Flops and parameters. Mo…
Image restoration is an important and challenging task in computer vision. Reverting a filtered image to its original image is helpful in various computer vision tasks. We employ a nonlinear activation function free network (NAFNet) for a fast and lightweight model and add a color attention module that extracts useful color information for better accuracy. We propose an accurate, fast, lightweight network with multi-scale and color attention for Instagram filter removal (CAIR). Experiment results show that the proposed CAIR outperforms existing Instagram filter removal networks in fast and lig…
University of Oulu（奥卢大学）；Tampere University（坦佩雷大学）；Czech Technical University（捷克技术大学）
We present LDP, a lightweight dense prediction neural architecture search (NAS) framework. Starting from a pre-defined generic backbone, LDP applies the novel Assisted Tabu Search for efficient architecture exploration. LDP is fast and suitable for various dense estimation problems, unlike previous NAS methods that are either computational demanding or deployed only for a single subtask. The performance of LPD is evaluated on monocular depth estimation, semantic segmentation, and image super-resolution tasks on diverse datasets, including NYU-Depth-v2, KITTI, Cityscapes, COCO-stuff, DIV2K, Set…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）
Real-time understanding in video is crucial in various AI applications such as autonomous driving. This work presents a fast single-shot segmentation strategy for video scene understanding. The proposed net, called S3-Net, quickly locates and segments target sub-scenes, meanwhile extracts structured time-series semantic features as inputs to an LSTM-based spatio-temporal model. Utilizing tensorization and quantization techniques, S3-Net is intended to be lightweight for edge computing. Experiments using CityScapes, UCF11, HMDB51 and MOMENTS datasets demonstrate that the proposed S3-Net achieve…
Remote photoplethysmography (rPPG) is an attractive camera-based health monitoring method that can measure the heart rhythm from facial videos. Many well-established deep-learning models have been reported to measure heart rate (HR) and heart rate variability (HRV). However, most of these models usually require a 30-second facial video and enormous computational resources to obtain accurate and robust results, which significantly limits their applications in real-world scenarios. Hence, we propose a lightweight pulse extraction network, FastBVP-Net, to quickly measure heart rhythm via facial v…
University of Electronic Science and Technology of China（电子科技大学）；Kyushu Institute of Technology（九州工业大学）
To achieve promising results on removing noise from real-world images, most of existing denoising networks are formulated with complex network structure, making them impractical for deployment. Some attempts focused on reducing the number of filters and feature channels but suffered from large performance loss, and a more practical and lightweight denoising network with fast inference speed is of high demand. To this end, a \textbf{Thu}mb\textbf{n}ail based \textbf{D}\textbf{e}noising Netwo\textbf{r}k dubbed Thunder, is proposed and implemented as a lightweight structure for fast restoration w…
Deep convolutional neural networks demonstrate impressive results in the super-resolution domain. A series of studies concentrate on improving peak signal noise ratio (PSNR) by using much deeper layers, which are not friendly to constrained resources. Pursuing a trade-off between the restoration capacity and the simplicity of models is still non-trivial. Recent contributions are struggling to manually maximize this balance, while our work achieves the same goal automatically with neural architecture search. Specifically, we handle super-resolution with a multi-objective approach. We also propo…
In the paper of ExquisiteNetV1, the ability of classification of ExquisiteNetV1 is worse than DenseNet. In this article, we propose a faster and better model ExquisiteNetV2. We conduct many experiments to evaluate its performance. We test ExquisiteNetV2, ExquisiteNetV1 and other 9 well-known models on 15 credible datasets under the same condition. According to the experimental results, ExquisiteNetV2 gets the highest classification accuracy over half of the datasets. Important of all, ExquisiteNetV2 has fewest amounts of parameters. Besides, in most instances, ExquisiteNetV2 has fastest comput…
Latency-critical computer vision systems, such as autonomous driving or drone control, require fast image or video compression when offloading neural network inference to a remote computer. To ensure low latency on a near-sensor edge device, we propose the use of lightweight encoders with constant bitrate and pruned encoding configurations, namely, ASTC and JPEG XS. Pruning introduces significant distortion which we show can be recovered by retraining the neural network with compressed data after decompression. Such an approach does not modify the network architecture or require coding format …
In this paper, we address the design of lightweight deep learning-based edge detection. The deep learning technology offers a significant improvement on the edge detection accuracy. However, typical neural network designs have very high model complexity, which prevents it from practical usage. In contrast, we propose a Fast Inference Network for Edge Detection (FINED), which is a lightweight neural net dedicated to edge detection. By carefully choosing proper components for edge detection purpose, we can achieve the state-of-the-art accuracy in edge detection while significantly reducing its c…
Pansharpening is a widely used image enhancement technique for remote sensing. Its principle is to fuse the input high-resolution single-channel panchromatic (PAN) image and low-resolution multi-spectral image and to obtain a high-resolution multi-spectral (HRMS) image. The existing deep learning pansharpening method has two shortcomings. First, features of two input images need to be concatenated along the channel dimension to reconstruct the HRMS image, which makes the importance of PAN images not prominent, and also leads to high computational cost. Second, the implicit information of featu…
Lightweight Data-Free Denoising for Detail-Preserving Biomedical Image Restoration
School of Computation, Information and Technology, Technical University of Munich（计算信息学院，慕尼黑技术大学）；Helmholtz AI, Helmholtz Munich - German Research Center for Environmental Health（海德堡人工智能，海德堡-德国环境健康研究所以及研究中心）；School of Biomedical Engineering and Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院）
Person Re-Identification aims to retrieve person identities from images captured by multiple cameras or the same cameras in different time instances and locations. Because of its importance in many vision applications from surveillance to human-machine interaction, person re-identification methods need to be reliable and fast. While more and more deep architectures are proposed for increasing performance, those methods also increase overall model complexity. This paper proposes a lightweight network that combines global, part-based, and channel features in a unified multi-branch architecture t…
FDDWNet: A Lightweight Convolutional Neural Network for Real-time Sementic Segmentation
1 National Engineering Research Center of Communications；Networking, Nanjing University of Posts \& Telecommunications, P.R. China. 2 School of Internet of Things, Nanjing University of Posts \& Telecommunications, P.R. China.
Camera relocalization involving a prior 3D reconstruction plays a crucial role in many mixed reality and robotics applications. Estimating the camera pose directly with respect to pre-built 3D models can be prohibitively expensive for several applications with limited storage and/or communication bandwidth. Although recent scene and absolute pose regression methods have become popular for efficient camera localization, most of them are computation-resource intensive and difficult to obtain a real-time inference with high accuracy constraints. This study proposes a simple scene regression metho…
National Yang Ming Chiao Tung University（台湾阳明交通大学）
This paper addresses fast semantic segmentation on video.Video segmentation often calls for real-time, or even fasterthan real-time, processing. One common recipe for conserving computation arising from feature extraction is to propagate features of few selected keyframes. However, recent advances in fast image segmentation make these solutions less attractive. To leverage fast image segmentation for furthering video segmentation, we propose a simple yet efficient propagation framework. Specifically, we perform lightweight flow estimation in 1/8-downscaled image space for temporal warping in s…
LGM-Pose: A Lightweight Global Modeling Network for Real-time Human Pose Estimation
School of Computer Science and Information Security（计算机科学与信息安全学院）；Guilin University of Electronic Technology（桂林电子科技大学）；Guangxi Metaverse Application Engineering Center（广西元宇宙应用工程中心）；Guangxi Key Laboratory of Trusted Software（广西可信软件重点实验室）；School of Electronic and Computer Engineering, Shenzhen Graduate School（电子与计算机工程学院，深圳研究生院）；Peking University（北京大学）
Existing face swap methods rely heavily on large-scale networks for adequate capacity to generate visually plausible results, which inhibits its applications on resource-constraint platforms. In this work, we propose MobileFSGAN, a novel lightweight GAN for face swap that can run on mobile devices with much fewer parameters while achieving competitive performance. A lightweight encoder-decoder structure is designed especially for image synthesis tasks, which is only 10.2MB and can run on mobile devices at a real-time speed. To tackle the unstability of training such a small network, we constru…
RePose: Learning Deep Kinematic Priors for Fast Human Pose Estimation
Google Inc.（谷歌公司）；University Of Waterloo（滑铁卢大学）
我们提出了一种新的高效和轻量级的模型，从一个单一的图像人体姿态估计。我们的模型的目的是实现竞争力的结果在一小部分的参数和计算成本的各种国家的最先进的方法。为此，我们明确地将基于部分的结构和几何先验的分层预测框架。在粗分辨率下，以类似于经典的基于部分的方法的方式，我们利用人体的运动学结构在关键点或身体部位之间传播卷积特征更新。与经典方法不同，我们采用端到端训练，通过数据的特征更新来学习这种几何先验。然后，我们以粗略的分辨率将特征表示传播到层次结构中，以粗略到精细的方式细化预测的姿势。最终的网络在轻量级深度神经网络中有效地对几何先验和直觉进行建模，在两个标准数据集Leeds Sports Pose和MPII Human Pose上为这种大小的模型产生最先进的结果。
Lightweight and Fast Real-time Image Enhancement via Decomposition of the Spatial-aware Lookup Tables
IPAI, Seoul National University, Seoul, Korea（IPAI，首尔国立大学，首尔，韩国）；LG Electronics, Seoul, Korea（LG电子，首尔，韩国）；Department of ECE, INMC, Seoul National University, Seoul, Korea（电子工程系，INMC，首尔国立大学，首尔，韩国）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Knowledge Distillation has been established as a highly promising approach for training compact and faster models by transferring knowledge from heavyweight and powerful models. However, KD in its conventional version constitutes an enduring, computationally and memory demanding process. In this paper, Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feat…
Lightweight Language-driven Grasp Detection using Conditional Consistency Model
FPT Software AI Center（FPT软件人工智能中心）；Automation & Control Institute, TU Wien（维也纳工业大学自动化与控制研究所）；Austrian Institute of Technology (AIT) GmbH（奥地利技术研究院）；Imperial College London（伦敦帝国学院）；University of Arkansas（阿肯色大学）；National University of Singapore（新加坡国立大学）；University of Liverpool（利物浦大学）
SINTEF（挪威科技工业研究院）；Bristol Royal Hospital for Children（布里斯托尔皇家儿童医院）；St. Olavs hospital（圣奥拉夫医院）
Automatic and consistent meningioma segmentation in T1-weighted MRI volumes and corresponding volumetric assessment is of use for diagnosis, treatment planning, and tumor growth evaluation. In this paper, we optimized the segmentation and processing speed performances using a large number of both surgically treated meningiomas and untreated meningiomas followed at the outpatient clinic. We studied two different 3D neural network architectures: (i) a simple encoder-decoder similar to a 3D U-Net, and (ii) a lightweight multi-scale architecture (PLS-Net). In addition, we studied the impact of dif…
Chongqing University of Science and Technology（重庆科技学院）
Real-time object detection on Unmanned Aerial Vehicles (UAVs) is a challenging issue due to the limited computing resources of edge GPU devices as Internet of Things (IoT) nodes. To solve this problem, in this paper, we propose a novel lightweight deep learning architectures named FasterX based on YOLOX model for real-time object detection on edge GPU. First, we design an effective and lightweight PixSF head to replace the original head of YOLOX to better detect small objects, which can be further embedded in the depthwise separable convolution (DS Conv) to achieve a lighter head. Then, a slim…
