In recent years, convolutional networks have demonstrated unprecedented performance in the image restoration task of super-resolution (SR). SR entails the upscaling of a single low-resolution image in order to meet application-specific image quality demands and plays a key role in mobile devices. To comply with privacy regulations and reduce the overhead of cloud computing, executing SR models locally on-device constitutes a key alternative approach. Nevertheless, the excessive compute and memory requirements of SR workloads pose a challenge in mapping SR networks on resource-constrained mobile platforms. This work presents MobiSR, a novel framework for performing efficient super-resolution on-device. Given a target mobile platform, the proposed framework considers popular model compression techniques and traverses the design space to reach the highest performing trade-off between image quality and processing speed. At run time, a novel scheduler dispatches incoming image patches to the appropriate model-engine pair based on the patch's estimated upscaling difficulty in order to meet the required image quality with minimum processing latency. Quantitative evaluation shows that the proposed framework yields on-device SR designs that achieve an average speedup of 2.13x over highly-optimized parallel difficulty-unaware mappings and 4.79x over highly-optimized single compute engine implementations.
Samsung AI Center（三星人工智能中心）；Rain Instant Pay（Rain即时支付公司）；University of Cambridge（剑桥大学）
In recent years, image and video delivery systems have begun integrating deep learning super-resolution (SR) approaches, leveraging their unprecedented visual enhancement capabilities while reducing reliance on networking conditions. Nevertheless, deploying these solutions on mobile devices still remains an active challenge as SR models are excessively demanding with respect to workload and memory footprint. Despite recent progress on on-device SR frameworks, existing systems either penalize visual quality, lead to excessive energy consumption or make inefficient use of the available resources…
Towards Lightest Low-Light Image Enhancement Architecture for Mobile Devices
Key Laboratory of Precision and Intelligent Chemistry, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China（精密与智能化学重点实验室，精密机械与精密仪器系，中国科学技术大学）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（信息与通信工程学院，电子科学与技术大学）
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；FRC CSC RAS（俄罗斯科学院联邦研究中心计算机科学研究所）；Smart Engines Service LLC（智能引擎服务有限责任公司）；Institute for Information Transmission Problems RAS（俄罗斯科学院信息传输问题研究所）
Quantized low-precision neural networks are very popular because they require less computational resources for inference and can provide high performance, which is vital for real-time and embedded recognition systems. However, their advantages are apparent for FPGA and ASIC devices, while general-purpose processor architectures are not always able to perform low-bit integer computations efficiently. The most frequently used low-precision neural network model for mobile central processors is an 8-bit quantized network. However, in a number of cases, it is possible to use fewer bits for weights …
We study performance characteristics of convolutional neural networks (CNN) for mobile computer vision systems. CNNs have proven to be a powerful and efficient approach to implement such systems. However, the system performance depends largely on the utilization of hardware accelerators, which are able to speed up the execution of the underlying mathematical operations tremendously through massive parallelism. Our contribution is performance characterization of multiple CNN-based models for object recognition and detection with several different hardware platforms and software frameworks, usin…
Bristol Robotics Laboratory, The University of Bristol（布里斯托尔机器人实验室，布里斯托尔大学）；Visual Information Laboratory, The University of Bristol（视觉信息实验室，布里斯托尔大学）；Department of Electrical and Electronic Engineering, The University of Manchester（曼彻斯特大学电气与电子工程系）；The Department of Computer Science and Engineering & The Department of Ophthalmology, The University of California, San Diego（加利福尼亚大学圣地亚哥分校计算机科学与工程系及眼科系）；Hangzhou ATG Intelligent Technology Co. Ltd.（杭州ATG智能科技有限公司）；Amazon（亚马逊公司）；The University of California, San Diego（加利福尼亚大学圣地亚哥分校）；The University of Manchester（曼彻斯特大学）；The University of Bristol（布里斯托尔大学）；Amazon, Seattle, USA（亚马逊公司（西雅图））；The Department of Ophthalmology, The University of California, San Diego（加利福尼亚大学圣地亚哥分校眼科系）
This paper presents an agile reactive navigation strategy for driving a non-holonomic ground vehicle around a preset course of gates in a cluttered environment using a low-cost processor array sensor. This enables machine vision tasks to be performed directly upon the sensor's image plane, rather than using a separate general-purpose computer. We demonstrate a small ground vehicle running through or avoiding multiple gates at high speed using minimal computational resources. To achieve this, target tracking algorithms are developed for the Pixel Processing Array and captured images are then pr…
In this paper, we present an OpenCL-based heterogeneous implementation of a computer vision algorithm -- image inpainting-based object removal algorithm -- on mobile devices. To take advantage of the computation power of the mobile processor, the algorithm workflow is partitioned between the CPU and the GPU based on the profiling results on mobile devices, so that the computationally-intensive kernels are accelerated by the mobile GPGPU (general-purpose computing using graphics processing units). By exploring the implementation trade-offs and utilizing the proposed optimization strategies at d…
School of Informatics（信息学院）；Xiamen University（厦门大学）
Low-quality face image restoration is a popular research direction in today's computer vision field. It can be used as a pre-work for tasks such as face detection and face recognition. At present, there is a lot of work to solve the problem of low-quality faces under various environmental conditions. This paper mainly focuses on the restoration of motion-blurred faces. In increasingly abundant mobile scenes, the fast recovery of motion-blurred faces can bring highly effective speed improvements in tasks such as face matching. In order to achieve this goal, a deblurring method for motion-blurre…
Peking University（北京大学）；IBM Research（IBM研究院）；Army Research Laboratory（陆军研究实验室）；Pennsylvania State University（宾夕法尼亚州立大学）
Convolutional Neural Networks (CNNs) have revolutionized the research in computer vision, due to their ability to capture complex patterns, resulting in high inference accuracies. However, the increasingly complex nature of these neural networks means that they are particularly suited for server computers with powerful GPUs. We envision that deep learning applications will be eventually and widely deployed on mobile devices, e.g., smartphones, self-driving cars, and drones. Therefore, in this paper, we aim to understand the resource requirements (time, memory) of CNNs on mobile devices. First,…
A recent trend in DNN development is to extend the reach of deep learning applications to platforms that are more resource and energy constrained, e.g., mobile devices. These endeavors aim to reduce the DNN model size and improve the hardware processing efficiency, and have resulted in DNNs that are much more compact in their structures and/or have high data sparsity. These compact or sparse models are different from the traditional large ones in that there is much more variation in their layer shapes and sizes, and often require specialized hardware to exploit sparsity for performance improve…
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute for Systems Analysis FRC CSC RAS（俄罗斯科学院联邦研究中心信息学与控制研究所系统分析研究所）；Smart Engines Service LLC（智能引擎服务有限责任公司）；Institute for Information Transmission Problems RAS（俄罗斯科学院信息传输问题研究所）
In this work we apply commonly known methods of non-adaptive interpolation (nearest pixel, bilinear, B-spline, bicubic, Hermite spline) and sampling (point sampling, supersampling, mip-map pre-filtering, rip-map pre-filtering and FAST) to the problem of projective image transformation. We compare their computational complexity, describe their artifacts and than experimentally measure their quality and working time on mobile processor with ARM architecture. Those methods were widely developed in the 90s and early 2000s, but were not in an area of active research in resent years due to a lower n…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Deep convolutional neural networks have dominated the pattern recognition scene by providing much more accurate solutions in computer vision problems such as object recognition and object detection. Most of these solutions come at a huge computational cost, requiring billions of multiply-accumulate operations and, thus, making their use quite challenging in real-time applications that run on embedded mobile (resource-power constrained) hardware. This work presents the architecture, the high-level synthesis design, and the implementation of SqueezeJet, an FPGA accelerator for the inference phas…
OPPO US Research Center InnoPeak Technology（OPPO 美国研究中心 InnoPeak Technology）
For many years, the family of convolutional neural networks (CNNs) has been a workhorse in deep learning. Recently, many novel CNN structures have been designed to address increasingly challenging tasks. To make them work efficiently on edge devices, researchers have proposed various structured network pruning strategies to reduce their memory and computational cost. However, most of them only focus on reducing the number of filter channels per layer without considering the redundancy within individual filter channels. In this work, we explore pruning from another dimension, the kernel size. W…
