UCMNet: Uncertainty-Aware Context Memory Network for Under-Display Camera Image Restoration
Hanyang University（汉阳大学）；Agency for Defense Development (ADD)（国防发展局（ADD））
Daehyun Kim, Youngmin Kim, Yoon Ju Oh, Tae Hyun Kim
备注：
We propose UCMNet, an uncertainty-aware adaptive framework that restores high-frequency details in regions with varying levels of degradation in under-display camera images
Under-display cameras (UDCs) allow for full-screen designs by positioning the imaging sensor underneath the display. Nonetheless, light diffraction and scattering through the various display layers result in spatially varying and complex degradations, which significantly reduce high-frequency details. Current PSF-based physical modeling techniques and frequency-separation networks are effective at reconstructing low-frequency structures and maintaining overall color consistency. However, they still face challenges in recovering fine details when dealing with complex, spatially varying degradation. To solve this problem, we propose a lightweight \textbf{U}ncertainty-aware \textbf{C}ontext-\textbf{M}emory \textbf{Network} (\textbf{UCMNet}), for UDC image restoration. Unlike previous methods that apply uniform restoration, UCMNet performs uncertainty-aware adaptive processing to restore high-frequency details in regions with varying degradations. The estimated uncertainty maps, learned through an uncertainty-driven loss, quantify spatial uncertainty induced by diffraction and scattering, and guide the Memory Bank to retrieve region-adaptive context from the Context Bank. This process enables effective modeling of the non-uniform degradation characteristics inherent to UDC imaging. Leveraging this uncertainty as a prior, UCMNet achieves state-of-the-art performance on multiple benchmarks with 30\% fewer parameters than previous models. Project page: \href{https://kdhrick2222.github.io/projects/UCMNet/}{https://kdhrick2222.github.io/projects/UCMNet}.
Under-display camera (UDC) is a novel technology that can make digital imaging experience in handheld devices seamless by providing large screen-to-body ratio. UDC images are severely degraded owing to their positioning under a display screen. This work addresses the restoration of images degraded as a result of UDC imaging. Two different networks are proposed for the restoration of images taken with two types of UDC technologies. The first method uses a pyramidal dilated convolution within a wavelet decomposed convolutional neural network for pentile-organic LED (P-OLED) based display system.…
The new trend of full-screen devices encourages us to position a camera behind a screen. Removing the bezel and centralizing the camera under the screen brings larger display-to-body ratio and enhances eye contact in video chat, but also causes image degradation. In this paper, we focus on a newly-defined Under-Display Camera (UDC), as a novel real-world single image restoration problem. First, we take a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED) and analyze their optical systems to understand the degradation. Second, we design a Monitor-Camera Imaging System (MCIS) for eas…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Under Display Cameras present a promising opportunity for phone manufacturers to achieve bezel-free displays by positioning the camera behind semi-transparent OLED screens. Unfortunately, such imaging systems suffer from severe image degradation due to light attenuation and diffraction effects. In this work, we present Deep Atrous Guided Filter (DAGF), a two-stage, end-to-end approach for image restoration in UDC systems. A Low-Resolution Network first restores image quality at low-resolution, which is subsequently used by the Guided Filter Network as a filtering input to produce a high-resolu…
This paper is the report of the first Under-Display Camera (UDC) image restoration challenge in conjunction with the RLQ workshop at ECCV 2020. The challenge is based on a newly-collected database of Under-Display Camera. The challenge tracks correspond to two types of display: a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED). Along with about 150 teams registered the challenge, eight and nine teams submitted the results during the testing phase for each track. The results in the paper are state-of-the-art restoration performance of Under-Display Camera Restoration. Datasets an…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Macau（澳门大学）；Shanghai AI Laboratory（上海人工智能实验室）
Under-Display Camera (UDC) has been widely exploited to help smartphones realize full screen display. However, as the screen could inevitably affect the light propagation process, the images captured by the UDC system usually contain flare, haze, blur, and noise. Particularly, flare and blur in UDC images could severely deteriorate the user experience in high dynamic range (HDR) scenes. In this paper, we propose a new deep model, namely UDC-UNet, to address the UDC image restoration problem with the known Point Spread Function (PSF) in HDR scenes. On the premise that Point Spread Function (PSF…
Nanyang Technological University（南洋理工大学）；SenseBrain；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). To bridge the gap, we introduce the first MIPI challenge including five tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Under-Display…
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Hong Kong University of Science and Technology（香港科技大学）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Samsung Research China - Beijing (SRC-B)（三星中国研究院北京分部）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳校区）；SAMSUNG Electronics（三星电子）
CSE, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；SISE, Shandong Normal University（山东师范大学信息科学与工程学院）
The exposed cameras of UAV can shake, shift, or even malfunction under the influence of harsh weather, while the add-on devices (Dupont lines) are very vulnerable to damage. We can place a low-cost T-OLED overlay around the camera to protect it, but this would also introduce image degradation issues. In particular, the temperature variations in the atmosphere can create mist that adsorbs to the T-OLED, which can cause secondary disasters (i.e., more severe image degradation) during the UAV's filming process. To solve the image degradation problem caused by overlaying T-OLEDs, in this paper we …
Under-display cameras have been proposed in recent years as a way to reduce the form factor of mobile devices while maximizing the screen area. Unfortunately, placing the camera behind the screen results in significant image distortions, including loss of contrast, blur, noise, color shift, scattering artifacts, and reduced light sensitivity. In this paper, we propose an image-restoration pipeline that is ISP-agnostic, i.e. it can be combined with any legacy ISP to produce a final image that matches the appearance of regular cameras using the same ISP. This is achieved with a deep learning app…
State Key Lab for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；Australian National University（澳大利亚国立大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Samsung Research China-Beijing (SRC-B)（三星中国研究院北京分部）；Imperial College London（伦敦帝国学院）
Under-display camera (UDC) provides an elegant solution for full-screen smartphones. However, UDC captured images suffer from severe degradation since sensors lie under the display. Although this issue can be tackled by image restoration networks, these networks require large-scale image pairs for training. To this end, we propose a modular network dubbed MPGNet trained using the generative adversarial network (GAN) framework for simulating UDC imaging. Specifically, we note that the UDC imaging degradation process contains brightness attenuation, blurring, and noise corruption. Thus we model …
Recent development of Under-Display Camera (UDC) systems provides a true bezel-less and notch-free viewing experience on smartphones (and TV, laptops, tablets), while allowing images to be captured from the selfie camera embedded underneath. In a typical UDC system, the microstructure of the semi-transparent organic light-emitting diode (OLED) pixel array attenuates and diffracts the incident light on the camera, resulting in significant image quality degradation. Oftentimes, noise, flare, haze, and blur can be observed in UDC images. In this work, we aim to analyze and tackle the aforemention…
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Peng Cheng Laboratory（鹏城实验室）
Image rescaling is a commonly used bidirectional operation, which first downscales high-resolution images to fit various display screens or to be storage- and bandwidth-friendly, and afterward upscales the corresponding low-resolution images to recover the original resolution or the details in the zoom-in images. However, the non-injective downscaling mapping discards high-frequency contents, leading to the ill-posed problem for the inverse restoration task. This can be abstracted as a general image degradation-restoration problem with information loss. In this work, we propose a novel inverti…
National University of Singapore（新加坡国立大学）；Wuhan Cobot Technology Co., Ltd.（武汉科博特科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Unwanted nonlinear gamma distortion frequently occurs in a great diversity of images during the procedures of image acquisition, processing, and/or display. And the gamma distortion often varies with capture setup change and luminance variation. Blind inverse gamma correction, which automatically determines a proper restoration gamma value from a given image, is of paramount importance to attenuate the distortion. For blind inverse gamma correction, an adaptive gamma transformation method (AGT-ME) is proposed directly from a maximized differential entropy model. And the corresponding optimizat…
Combined Flicker-banding and Moire Removal for Screen-Captured Images
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Central Media Technology Institute, Huawei, China（华为中央媒体技术研究所）；Department of XXX, University of YYY, Location, Country（YYY大学XXX系）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院WWW研究所）
From Fog Chamber to Aircraft Window: Pixel-Registered Imaging and Synthetic Fine-Tuning Enable Cross-Domain Defogging
Department of Electrical and Computer Engineering, University of Utah（犹他大学电气与计算机工程系）；U.S. Naval Research Laboratory Remote Sensing Division（美国海军研究实验室远程传感部门）
UDC-VIT: A Real-World Video Dataset for Under-Display Cameras
Dept. of Data Science, Seoul National University（数据科学系，首尔国立大学）；Dept. of Computer Science and Engineering, Seoul National University（计算机科学与工程系，首尔国立大学）；Research Center, Samsung Display Co., Ltd.（三星显示公司研究部）；Mobile Display Electronics Development Team, Samsung Display Co., Ltd.（三星显示公司移动显示电子开发团队）
Instituto de Matemática e Estatística, Universidade de São Paulo（圣保罗大学数学与统计研究所）；Universidade Federal de São Carlos（圣卡洛斯联邦大学）；Mayo Clinic College of Medicine（梅奥诊所医学院）；Universidade Federal de Alagoas（阿拉戈斯联邦大学）
Vibro-acoustography (VA) is a medical imaging method based on the difference-frequency generation produced by the mixture of two focused ultrasound beams. VA has been applied to different problems in medical imaging such as imaging bones, microcalcifications in the breast, mass lesions, and calcified arteries. The obtained images may have a resolution of 0.7--0.8 mm. Current VA systems based on confocal or linear array transducers generate C-scan images at the beam focal plane. Images on the axial plane are also possible, however the system resolution along depth worsens when compared to the l…
School of Informatics（信息学院）；Xiamen University（厦门大学）
Low-quality face image restoration is a popular research direction in today's computer vision field. It can be used as a pre-work for tasks such as face detection and face recognition. At present, there is a lot of work to solve the problem of low-quality faces under various environmental conditions. This paper mainly focuses on the restoration of motion-blurred faces. In increasingly abundant mobile scenes, the fast recovery of motion-blurred faces can bring highly effective speed improvements in tasks such as face matching. In order to achieve this goal, a deblurring method for motion-blurre…
MoiréNet: A Compact Dual-Domain Network for Image Demoiréing
School of Future Science and Engineering, Soochow University, China（未来科学与工程学院，苏州大学）；Clobotics, China（中国Clobotics公司）；Fraunhofer Institute for Applied Information Technology, Germany（德国应用信息技术研究所）；School of Mathematical and Computer Sciences, Heriot-Watt University (Malaysia Campus), Malaysia（数学与计算机科学学院，赫瑞-瓦特大学（马来西亚校区））
Bit-depth is the number of bits for each color channel of a pixel in an image. Although many modern displays support unprecedented higher bit-depth to show more realistic and natural colors with a high dynamic range, most media sources are still in bit-depth of 8 or lower. Since insufficient bit-depth may generate annoying false contours or lose detailed visual appearance, bit-depth expansion (BDE) from low bit-depth (LBD) images to high bit-depth (HBD) images becomes more and more important. In this paper, we adopt a learning-based approach for BDE and propose a novel CNN-based bit-depth expa…
Learning Weighting Map for Bit-Depth Expansion within a Rational Range
School of Software, Dalian University of Technology（大连理工大学软件学院）；International School of Information Science and Engineering, Dalian University of Technology（大连理工大学国际信息与工程学院）；School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）
