We review camera architecture in the age of artificial intelligence. Modern cameras use physical components and software to capture, compress and display image data. Over the past 5 years, deep learning solutions have become superior to traditional algorithms for each of these functions. Deep learning enables 10-100x reduction in electrical sensor power per pixel, 10x improvement in depth of field and dynamic range and 10-100x improvement in image pixel count. Deep learning enables multiframe and multiaperture solutions that fundamentally shift the goals of physical camera design. Here we review the state of the art of deep learning in camera operations and consider the impact of AI on the physical design of cameras.
We assess the accuracy of a smartphone camera simulation. The simulation is an end-to-end analysis that begins with a physical description of a high dynamic range 3D scene and includes a specification of the optics and the image sensor. The simulation is compared to measurements of a physical version of the scene. The image system simulation accurately matched measurements of optical blur, depth of field, spectral quantum efficiency, scene inter-reflections, and sensor noise. The results support the use of image systems simulation methods for soft prototyping cameras and for producing syntheti…
In this paper we reverse engineer the Sony IMX219PQ image sensor, otherwise known as the Raspberry Pi Camera v2.0. We provide a visual reference for pixel non-uniformity by analysing variations in transistor length, microlens optic system and in the photodiode. We use these measurements to demonstrate irregularities at the microscopic level and link this to the signal variation measured as pixel non-uniformity used for unique identification of discrete image sensors.
Smartphone is the superstar product in digital device market and the quality of smartphone camera photos (SCPs) is becoming one of the dominant considerations when consumers purchase smartphones. How to evaluate the quality of smartphone cameras and the taken photos is urgent issue to be solved. To bridge the gap between academic research accomplishment and industrial needs, in this paper, we establish a new Smartphone Camera Photo Quality Database (SCPQD2020) including 1800 images with 120 scenes taken by 15 smartphones. Exposure, color, noise and texture which are four dominant factors influ…
atlanTTic research center for Telecommunication Technologies, University of Vigo（维戈大学 atlanTTic 电信技术研究中心）
This communication is about an application of image forensics where we use camera sensor fingerprints to identify source camera (SCI: Source Camera Identification) in webcam/smartphone videos. Sensor or camera fingerprints are based on computing the intrinsic noise that is always present in this kind of sensors due to manufacturing imperfections. This is an unavoidable characteristic that links each sensor with its noise pattern. PRNU (Photo Response Non-Uniformity) has become the default technique to compute a camera fingerprint. There are many applications nowadays dealing with PRNU patterns…
The principle of Photo Response Non Uniformity (PRNU) is often exploited to deduce the identity of the smartphone device whose camera or sensor was used to acquire a certain image. In this work, we design an algorithm that perturbs a face image acquired using a smartphone camera such that (a) sensor-specific details pertaining to the smartphone camera are suppressed (sensor anonymization); (b) the sensor pattern of a different device is incorporated (sensor spoofing); and (c) biometric matching using the perturbed image is not affected (biometric utility). We employ a simple approach utilizing…
We present a technique called single-pixel imaging camera based on complementary detection and optimized Bernoulli modulation (CSPI camera), which can significantly reduce the influence of the disturbance light to single-pixel imaging (SPI). The numerical simulation demonstrates that when the probability of the value "1" is P=0.5 for each coded pattern, CSPI camera is disturbance-free is still disturbance-free even if the intensity fluctuation of the disturbance light is much larger than the signal's intensity. The reconstruction results of both SPI and differential SPI are also compared. This…
This study focuses on assessing smartphone camera characteristics for developing an economic smartphone-based Particle Image Velocimetry (PIV) system. In the investigation, flow around a cylinder was visualized using two commercially-available smartphones (OnePlus 5T and iPhone X) cameras and low-intensity laser diodes. Hydrogen bubbles generated from electrolysis (termed Bubble Image Velocimetry) of aluminum electrodes were used as seeding medium. OpenPIV, an open-source toolbox, was used for processing captured images and obtaining the flow fields. A parametric analysis of the two smartphone…
To compose a 360 image from a rig with multiple fisheye cameras, a conventional processing pipeline first performs demosaicking on each fisheye camera's Bayer-patterned grid, then translates demosaicked pixels from the camera grid to a rectified image grid---thus performing two image interpolation steps in sequence. Hence interpolation errors can accumulate, and acquisition noise in the captured pixels can pollute neighbors in two consecutive processing stages. In this paper, we propose a joint processing framework that performs demosaicking and grid-to-grid mapping simultaneously---thus limit…
In this work we have determined the focal length of a concave lens by photographing the virtual image of an object by a smartphone camera. We have similarly determined the focal length of a convex lens by forming a virtual image of an object keeping it within the focal distance from the lens. When a photograph is taken by a smartphone, the transverse width of the image on the sensor of the camera in pixels can be read off by software available freely from the internet. By taking a photograph of the virtual image from two positions of the camera separated by a distance along the line of sight o…
We present a compact, diffuser-assisted, single-pixel computational camera. A rotating ground glass diffuser is adopted, in preference to a commonly used digital micro-mirror device (DMD), to encode a two-dimensional (2D) image into single-pixel signals. We retrieve images with an 8.8% sampling ratio after the calibration of the pseudo-random pattern of the diffuser under incoherent illumination. Furthermore, we demonstrate hyperspectral imaging with line array detection by adding a diffraction grating. The implementation results in a cost-effective single-pixel camera for high-dimensional ima…
Computational ghost imaging (CGI) and single-pixel camera (SPC) are two types of single-pixel imaging and attracts increasing interest in recent years. The performance differences of CGI and SPC in difference light disturbance environment are investigated. In comparison with CGI, we demonstrate that the quality of SPC is always better in the same conditions and local light disturbance has a larger influence to CGI/SPC than global light disturbance. In addition, a method to improve the reconstruction result of SPC is proposed if the source's energy is unstable and there is light disturbance in …
University of Southern California（南加利福尼亚大学）；Information Sciences Institute（信息科学研究所）
Hyperspectral cameras generate a large amount of data due to the presence of hundreds of spectral bands as opposed to only three channels (red, green, and blue) in traditional cameras. This requires a significant amount of data transmission between the hyperspectral image sensor and a processor used to classify/detect/track the images, frame by frame, expending high energy and causing bandwidth and security bottlenecks. To mitigate this problem, we propose a form of processing-in-pixel (PIP) that leverages advanced CMOS technologies to enable the pixel array to perform a wide range of complex …
University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Time-of-flight (ToF) cameras are becoming increasingly popular for 3D imaging. Their optimal usage has been studied from the several aspects. One of the open research problems is the possibility of a multicamera interference problem when two or more ToF cameras are operating simultaneously. In this work we present an efficient method to synchronize multiple operating ToF cameras. Our method is based on the time-division multiplexing, but unlike traditional time multiplexing, it does not decrease the effective camera frame rate. Additionally, for unsynchronized cameras, we provide a robust meth…
Real time, high resolution 3D reconstruction of scenes hidden from the direct field of view is a challenging field of research with applications in real-life situations related e.g. to surveillance, self-driving cars and rescue missions. Most current techniques recover the 3D structure of a non-lineof-sight (NLOS) static scene by detecting the return signal from the hidden object on a scattering observation area. Here, we demonstrate the full colour retrieval of the 3D shape of a hidden scene by coupling back-projection imaging algorithms with the high-resolution time-of-flight information pro…
We present a feature-free photogrammetric technique that enables quantitative 3D mesoscopic (mm-scale height variation) imaging with tens-of-micron accuracy from sequences of images acquired by a smartphone at close range (several cm) under freehand motion without additional hardware. Our end-to-end, pixel-intensity-based approach jointly registers and stitches all the images by estimating a coaligned height map, which acts as a pixel-wise radial deformation field that orthorectifies each camera image to allow homographic registration. The height maps themselves are reparameterized as the outp…
University College London（伦敦大学学院）；Mullard Space Science Laboratory（穆拉德空间科学实验室）
This paper investigates how the inherent quantization of camera sensors introduces uncertainty in the calculated position of an observed feature during 3-D mapping. It is typically assumed that pixels and scene features are points, however, a pixel is a two-dimensional area that maps onto multiple points in the scene. This uncertainty region is a bound for quantization error in the calculated point positions. Earlier studies calculated the volume of two intersecting pixel views, approximated as a cuboid, by projecting pyramids and cones from the pixels into the scene. In this paper, we reverse…
High-dynamic-range (HDR) photography involves fusing a bracket of images taken at different exposure settings in order to compensate for the low dynamic range of digital cameras such as the ones used in smartphones. In this paper, a method for automatically selecting the exposure settings of such images is introduced based on the camera characteristic function. In addition, a new fusion method is introduced based on an optimization formulation and weighted averaging. Both of these methods are implemented on a smartphone platform as an HDR app to demonstrate the practicality of the introduced m…
The quality of images captured by smartphones is an important specification since smartphones are becoming ubiquitous as primary capturing devices. The traditional image signal processing (ISP) pipeline in a smartphone camera consists of several image processing steps performed sequentially to reconstruct a high quality sRGB image from the raw sensor data. These steps consist of demosaicing, denoising, white balancing, gamma correction, colour enhancement, etc. Since each of them are performed sequentially using hand-crafted algorithms, the residual error from each processing module accumulate…
