Critical aspects of computational imaging systems, such as experimental design and image priors, can be optimized through deep networks formed by the unrolled iterations of classical model-based reconstructions (termed physics-based networks). However, for real-world large-scale inverse problems, computing gradients via backpropagation is infeasible due to memory limitations of graphics processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable data-driven design for large-scale computational imaging systems. We demonstrate our method on a small-scale compressed sensing example, as well as two large-scale real-world systems: multi-channel magnetic resonance imaging and super-resolution optical microscopy.
Computational imaging has been playing a vital role in the development of natural sciences. Advances in sensory, information, and computer technologies have further extended the scope of influence of imaging, making digital images an essential component of our daily lives. For the past three decades, we have witnessed phenomenal developments of mathematical and machine learning methods in computational imaging. In this note, we will review some of the recent developments of the machine learning approach for computational imaging and discuss its differences and relations to the mathematical app…
Machine learning methods for computational imaging require uncertainty estimation to be reliable in real settings. While Bayesian models offer a computationally tractable way of recovering uncertainty, they need large data volumes to be trained, which in imaging applications implicates prohibitively expensive collections with specific imaging instruments. This paper introduces a novel framework to train variational inference for inverse problems exploiting in combination few experimentally collected data, domain expertise and existing image data sets. In such a way, Bayesian machine learning m…
Computational image reconstruction algorithms generally produce a single image without any measure of uncertainty or confidence. Regularized Maximum Likelihood (RML) and feed-forward deep learning approaches for inverse problems typically focus on recovering a point estimate. This is a serious limitation when working with underdetermined imaging systems, where it is conceivable that multiple image modes would be consistent with the measured data. Characterizing the space of probable images that explain the observational data is therefore crucial. In this paper, we propose a variational deep pr…
University of Kent（肯特大学）；Nankai University（南开大学）；Shandong University of Technology（山东理工大学）；University of Haifa（海法大学）
Recent work showed neural-network-based approaches to reconstructing images from compressively sensed measurements offer significant improvements in accuracy and signal compression. Such methods can dramatically boost the capability of computational imaging hardware. However, to date, there have been two major drawbacks: (1) the high-precision real-valued sensing patterns proposed in the majority of existing works can prove problematic when used with computational imaging hardware such as a digital micromirror sampling device and (2) the network structures for image reconstruction involve inte…
Lensless imaging seeks to replace/remove the lens in a conventional imaging system. The earliest cameras were in fact lensless, relying on long exposure times to form images on the other end of a small aperture in a darkened room/container (camera obscura). The introduction of a lens allowed for more light throughput and therefore shorter exposure times, while retaining sharp focus. The incorporation of digital sensors readily enabled the use of computational imaging techniques to post-process and enhance raw images (e.g. via deblurring, inpainting, denoising, sharpening). Recently, imaging sc…
Massachusetts Institute of Technology（麻省理工学院）；Singapore-MIT Alliance for Research and Technology (SMART) Centre（新加坡-麻省理工学院科研与技术联盟（SMART）中心）
Deep learning has been proven to yield reliably generalizable answers to numerous classification and decision tasks. Here, we demonstrate for the first time, to our knowledge, that deep neural networks (DNNs) can be trained to solve inverse problems in computational imaging. We experimentally demonstrate a lens-less imaging system where a DNN was trained to recover a phase object given a raw intensity image recorded some distance away.
Brain-computer interfaces (BCIs) are enabling a range of new possibilities and routes for augmenting human capability. Here, we propose BCIs as a route towards forms of computation, i.e. computational imaging, that blend the brain with external silicon processing. We demonstrate ghost imaging of a hidden scene using the human visual system that is combined with an adaptive computational imaging scheme. This is achieved through a projection pattern `carving' technique that relies on real-time feedback from the brain to modify patterns at the light projector, thus enabling more efficient and hig…
Beijing Institute of Technology（北京理工大学）；Westlake University（西湖大学）；University of Arizona（亚利桑那大学）
Computational reconstruction plays a vital role in computer vision and computational photography. Most of the conventional optimization and deep learning techniques explore local information for reconstruction. Recently, nonlocal low-rank (NLR) reconstruction has achieved remarkable success in improving accuracy and generalization. However, the computational cost has inhibited NLR from seeking global structural similarity, which consequentially keeps it trapped in the tradeoff between accuracy and efficiency and prevents it from high-dimensional large-scale tasks. To address this challenge, we…
Rice University（莱斯大学）；Indian Institute of Technology (IIT) Madras（印度理工学院马德拉斯分校）；Insight Centre for Data Analytics（数据 analytics 洞察中心）
Signal reconstruction is a challenging aspect of computational imaging as it often involves solving ill-posed inverse problems. Recently, deep feed-forward neural networks have led to state-of-the-art results in solving various inverse imaging problems. However, being task specific, these networks have to be learned for each inverse problem. On the other hand, a more flexible approach would be to learn a deep generative model once and then use it as a signal prior for solving various inverse problems. We show that among the various state of the art deep generative models, autoregressive models…
Computational Imaging for Machine Perception: Transferring Semantic Segmentation beyond Aberrations
Zhejiang University（浙江大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hunan University（湖南大学）；Shanghai SUPREMIND Technology Company Ltd（上海超脑科技有限公司）；Hangzhou SurImage Technology Company Ltd（杭州 SurImage 科技有限公司）
University of California Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute（加州纳米系统研究所）
We show that dispersive propagation of light followed by phase detection has properties that can be exploited for extracting features from the waveforms. This discovery is spearheading development of a new class of physics-inspired algorithms for feature extraction from digital images with unique properties and superior dynamic range compared to conventional algorithms. In certain cases, these algorithms have the potential to be an energy efficient and scalable substitute to synthetically fashioned computational techniques in practice today.
Computational compressive imaging reconstruction with a simulated-annealing-based Ising solver
Institute for Quantum Sensing and Information Processing, State Key Laboratory of Photonics and Communications, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Hefei National Laboratory（合肥国家实验室）；Shanghai Research Center for Quantum Sciences（上海量子科学研究与探索中心）
Zhejiang University（浙江大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；ETH Zürich（苏黎世联邦理工学院）
Panoramic Annular Lens (PAL) composed of few lenses has great potential in panoramic surrounding sensing tasks for mobile and wearable devices because of its tiny size and large Field of View (FoV). However, the image quality of tiny-volume PAL confines to optical limit due to the lack of lenses for aberration correction. In this paper, we propose an Annular Computational Imaging (ACI) framework to break the optical limit of light-weight PAL design. To facilitate learning-based image restoration, we introduce a wave-based simulation pipeline for panoramic imaging and tackle the synthetic-to-re…
Optical Linear Systems Framework for Event Sensing and Computational Neuromorphic Imaging
International Centre for Neuromorphic Systems, The MARCS Institute, Western Sydney University（西悉尼大学MARCS研究所国际神经形态系统中心）；Centre for Research in Mathematics and Data Science, Western Sydney University（西悉尼大学数学与数据科学研究中心）
Fast and Memory-efficient Non-line-of-sight Imaging with Quasi-Fresnel Transform
State Key Laboratory of Precision Space-time Information Sensing Technology（精密时空信息感知技术国家重点实验室）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（燕琪湖北京应用数学科学研究院）
Computational ghost imaging is a promising technique for single-pixel imaging because it is robust to disturbance and can be operated over broad wavelength bands, unlike common cameras. However, one disadvantage of this method is that it has a long calculation time for image reconstruction. In this paper, we have designed a dedicated calculation circuit that accelerated the process of computational ghost imaging. We implemented this circuit by using a field-programmable gate array, which reduced the calculation time for the circuit compared to a CPU. The dedicated circuit reconstructs images a…
Analog In-Memory Computing with Uncertainty Quantification for Efficient Edge-based Medical Imaging Segmentation
École nationale Supérieure d’Informatique（国立高等计算机学院）；IBM Research Europe（IBM欧洲研究院）；Yorktown High school（约克镇高中）；IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）
We have designed a single-pixel camera with imaging around corners based on computational ghost imaging. It can obtain the image of an object when the camera cannot look at the object directly. Our imaging system explores the fact that a bucket detector in a ghost imaging setup has no spatial resolution capability. A series of experiments have been designed to confirm our predictions. This camera has potential applications for imaging around corner or other similar environments where the object cannot be observed directly.
Technical University of Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）；University Hospital of Tuebingen（图宾根大学医院）；University of Minnesota（明尼苏达大学）；New York University（纽约大学）；Friedrich-Alexander University Erlangen（埃尔朗根-纽伦堡大学）
Physics-driven deep learning methods have emerged as a powerful tool for computational magnetic resonance imaging (MRI) problems, pushing reconstruction performance to new limits. This article provides an overview of the recent developments in incorporating physics information into learning-based MRI reconstruction. We consider inverse problems with both linear and non-linear forward models for computational MRI, and review the classical approaches for solving these. We then focus on physics-driven deep learning approaches, covering physics-driven loss functions, plug-and-play methods, generat…
U-Nets have been established as a standard architecture for image-to-image learning problems such as segmentation and inverse problems in imaging. For large-scale data, as it for example appears in 3D medical imaging, the U-Net however has prohibitive memory requirements. Here, we present a new fully-invertible U-Net-based architecture called the iUNet, which employs novel learnable and invertible up- and downsampling operations, thereby making the use of memory-efficient backpropagation possible. This allows us to train deeper and larger networks in practice, under the same GPU memory restric…
Nanjing University of Science and Technology（南京理工大学）
Strong scattering medium brings great difficulties to optical imaging, which is also a problem in medical imaging and many other fields. Optical memory effect makes it possible to image through strong random scattering medium. However, this method also has the limitation of limited angle field-of-view (FOV), which prevents it from being applied in practice. In this paper, a kind of practical convolutional neural network called PDSNet is proposed, which effectively breaks through the limitation of optical memory effect on FOV. Experiments is conducted to prove that the scattered pattern can be …
Charles Stark Draper Laboratory（查尔斯·斯塔克·德拉普尔实验室）；Stanford Medicine（斯坦福医学）；Bracco Diagnostics Corporation（布拉科诊断公司）
Molecularly targeted contrast enhanced ultrasound (mCEUS) is a clinically promising approach for early cancer detection through targeted imaging of VEGFR2 (KDR) receptors. We have developed computational enhancement techniques for mCEUS tailored to address the unique challenges of imaging contrast accumulation in humans. These techniques utilize dynamic analysis to distinguish molecularly bound contrast agent from other contrast-mode signal sources, enabling analysis of contrast agent accumulation to be performed during contrast bolus arrival when the signal due to molecular binding is stronge…
The unpaired training can be the only option available for fast deep learning-based ghost imaging, where obtaining a high signal-to-noise ratio (SNR) image copy of each low SNR ghost image could be practically time-consuming and challenging. This paper explores the capabilities of deep learning to leverage computational ghost imaging when there is a lack of paired training images. The deep learning approach proposed here enables fast ghost imaging through reconstruction of high SNR images from faint and hastily shot ghost images using a constrained Wasserstein generative adversarial network. I…
High dynamic range (HDR) imaging is still a challenging task in modern digital photography. Recent research proposes solutions that provide high-quality acquisition but at the cost of a very large number of operations and a slow inference time that prevent the implementation of these solutions on lightweight real-time systems. In this paper, we propose CEN-HDR, a new computationally efficient neural network by providing a novel architecture based on a light attention mechanism and sub-pixel convolution operations for real-time HDR imaging. We also provide an efficient training scheme by applyi…
Chiba University（千叶大学）；Kanazawa University（金泽大学）；Mitsubishi Electric Corporation（三菱电机株式会社）
Computational ghost imaging (CGI) is a single-pixel imaging technique that exploits the correlation between known random patterns and the measured intensity of light transmitted (or reflected) by an object. Although CGI can obtain two- or three- dimensional images with a single or a few bucket detectors, the quality of the reconstructed images is reduced by noise due to the reconstruction of images from random patterns. In this study, we improve the quality of CGI images using deep learning. A deep neural network is used to automatically learn the features of noise-contaminated CGI images. Aft…
Compressive sensing is considered a huge breakthrough in signal acquisition. It allows recording an image consisting of $N^2$ pixels using much fewer than $N^2$ measurements if it can be transformed to a basis where most pixels take on negligibly small values. Standard compressive sensing techniques suffer from the computational overhead needed to reconstruct an image with typical computation times between hours and days and are thus not optimal for applications in physics and spectroscopy. We demonstrate an adaptive compressive sampling technique that performs measurements directly in a spars…
We present a proof-of-concept end-to-end system for computational extended depth of field (EDOF) imaging. The acquisition is performed through a phase-coded aperture implemented by placing a thin wavelength-dependent optical mask inside the pupil of a conventional camera lens, as a result of which, each color channel is focused at a different depth. The reconstruction process receives the raw Bayer image as the input, and performs blind estimation of the output color image in focus at an extended range of depths using a patch-wise sparse prior. We present a fast non-iterative reconstruction al…
Center for Biomedical Image Computing and Analytics（生物医学图像计算与分析中心）；Perelman School of Medicine（佩雷尔曼医学院）；University of Pennsylvania（宾夕法尼亚大学）
Decoding brain functional states underlying different cognitive processes using multivariate pattern recognition techniques has attracted increasing interests in brain imaging studies. Promising performance has been achieved using brain functional connectivity or brain activation signatures for a variety of brain decoding tasks. However, most of existing studies have built decoding models upon features extracted from imaging data at individual time points or temporal windows with a fixed interval, which might not be optimal across different cognitive processes due to varying temporal durations…
Single-Photon Avalanche Diodes (SPAD) are affordable photodetectors, capable to collect extremely fast low-energy events, due to their single-photon sensibility. This makes them very suitable for time-of-flight-based range imaging systems, allowing to reduce costs and power requirements, without sacrifizing much temporal resolution. In this work we describe a computational model to simulate the behaviour of SPAD sensors, aiming to provide a realistic camera model for time-resolved light transport simulation, with applications on prototyping new reconstructions techniques based on SPAD time-of-…
