Recently, image enhancement and restoration have become important applications on mobile devices, such as super-resolution and image deblurring. However, most state-of-the-art networks present extremely high computational complexity. This makes them difficult to be deployed on mobile devices with acceptable latency. Moreover, when deploying to different mobile devices, there is a large latency variation due to the difference and limitation of deep learning accelerators on mobile devices. In this paper, we conduct a search of portable network architectures for better quality-latency trade-off across mobile devices. We further present the effectiveness of widely used network optimizations for image deblurring task. This paper provides comprehensive experiments and comparisons to uncover the in-depth analysis for both latency and image quality. Through all the above works, we demonstrate the successful deployment of image deblurring application on mobile devices with the acceleration of deep learning accelerators. To the best of our knowledge, this is the first paper that addresses all the deployment issues of image deblurring task across mobile devices. This paper provides practical deployment-guidelines, and is adopted by the championship-winning team in NTIRE 2020 Image Deblurring Challenge on Smartphone Track.
We propose fixed-function neural network hardware that is designed to perform pixel-to-pixel image transformations in a highly efficient way. We use a fully trainable, fixed-topology neural network to build a model that can perform a wide variety of image processing tasks. Our model uses compressed skip lines and hybrid FIR-IIR blocks to reduce the latency and hardware footprint. Our proposed Machine Learning Imaging Core, dubbed MagIC, uses a silicon area of ~3mm^2 (in TSMC 16nm), which is orders of magnitude smaller than a comparable pixel-wise dense prediction model. MagIC requires no DDR b…
Joint Transmission and Deblurring: A Semantic Communication Approach Using Events
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）
Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos. To this end, we present a unified framework of event-based motion deblurring and frame interpolation for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction. Specifically, the mapping relation between blurry frames and sharp latent images is first predicted by a learnable double integral network, and a fusion network is …
Microsoft Research（微软研究院）；Tsinghua University（清华大学）
Neural Architecture Search (NAS) has shown promising performance in the automatic design of vision transformers (ViT) exceeding 1G FLOPs. However, designing lightweight and low-latency ViT models for diverse mobile devices remains a big challenge. In this work, we propose ElasticViT, a two-stage NAS approach that trains a high-quality ViT supernet over a very large search space that supports a wide range of mobile devices, and then searches an optimal sub-network (subnet) for direct deployment. However, prior supernet training methods that rely on uniform sampling suffer from the gradient conf…
We present a proof-of-concept end-to-end system for computational extended depth of field (EDOF) imaging. The acquisition is performed through a phase-coded aperture implemented by placing a thin wavelength-dependent optical mask inside the pupil of a conventional camera lens, as a result of which, each color channel is focused at a different depth. The reconstruction process receives the raw Bayer image as the input, and performs blind estimation of the output color image in focus at an extended range of depths using a patch-wise sparse prior. We present a fast non-iterative reconstruction al…
Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference …
Deploying and Evaluating Multiple Deep Learning Models on Edge Devices for Diabetic Retinopathy Detection
Ghana Communication Technology University（加纳通信技术大学）；Council for Scientific and Industrial Research（科学与工业研究委员会）；Shenzhen Technology University（深圳技术大学）；Kwame Nkrumah University of Science and Technology（夸梅·恩克鲁玛科技大学）
MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a cloud deployment due to uncertainties in resource allocations across many virtual instances. We propose the collage inference technique which uses a novel convolutional neural network model, collage-cnn, to provide low-cost redundancy. A collage-cnn model takes a collage image f…
ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers
School of Computing Science, Simon Fraser University（计算科学系，西蒙·弗雷泽大学）；Global College, Shanghai Jiao Tong University（全球学院，上海交通大学）；School of Science and Engineering (SSE) and The Future Network of Intelligence Institute (FNii), The Chinese University of Hong Kong, Shenzhen（科学与工程学院（SSE）和未来智能网络研究所（FNii），香港中文大学（深圳））；Computer Science Department, University of Illinois Urbana-Champaign（计算机科学系，伊利诺伊大学厄巴纳-香槟分校）
European Organization for Nuclear Research (CERN)（欧洲核子研究组织）；Institute of Physics Belgrade（贝尔格莱德物理研究所）；Politecnico di Milano（米兰理工大学）；California Institute of Technology（加州理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；Zenseact（Zenseact公司）；ICEPP, University of Tokyo（东京大学国际基本粒子物理研究所）；Columbia University（哥伦比亚大学）；University of California San Diego（加州大学圣地亚哥分校）；Massachusetts Institute of Technology（麻省理工学院）；Fermi National Accelerator Laboratory（费米国家加速器实验室）；Purdue University（普渡大学）；HawkEye360（鹰眼360公司）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Rhodes College（罗德学院）
We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on FPGAs. By extending the hls4ml library, we demonstrate an inference latency of $5\,μ$s using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detec…
