Video frame interpolation (VFI) enables many important applications that might involve the temporal domain, such as slow motion playback, or the spatial domain, such as stop motion sequences. We are focusing on the former task, where one of the key challenges is handling high dynamic range (HDR) scenes in the presence of complex motion. To this end, we explore possible advantages of dual-exposure sensors that readily provide sharp short and blurry long exposures that are spatially registered and whose ends are temporally aligned. This way, motion blur registers temporally continuous information on the scene motion that, combined with the sharp reference, enables more precise motion sampling within a single camera shot. We demonstrate that this facilitates a more complex motion reconstruction in the VFI task, as well as HDR frame reconstruction that so far has been considered only for the originally captured frames, not in-between interpolated frames. We design a neural network trained in these tasks that clearly outperforms existing solutions. We also propose a metric for scene motion complexity that provides important insights into the performance of VFI methods at the test time.
Video frame interpolation, which aims to synthesize non-exist intermediate frames in a video sequence, is an important research topic in computer vision. Existing video frame interpolation methods have achieved remarkable results under specific assumptions, such as instant or known exposure time. However, in complicated real-world situations, the temporal priors of videos, i.e. frames per second (FPS) and frame exposure time, may vary from different camera sensors. When test videos are taken under different exposure settings from training ones, the interpolated frames will suffer significant m…
Harbin Institute of Technology（哈尔滨工业大学）；City University of Hong Kong（香港城市大学）；Peng Cheng Laboratory（鹏城实验室）
Natural videos captured by consumer cameras often suffer from low framerate and motion blur due to the combination of dynamic scene complexity, lens and sensor imperfection, and less than ideal exposure setting. As a result, computational methods that jointly perform video frame interpolation and deblurring begin to emerge with the unrealistic assumption that the exposure time is known and fixed. In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame interpolation and deblurring under unknown exposure time. Toward this goal, we first adopt a varian…
Wuhan University of Science and Technology（武汉科技大学）；Institute for Infocomm Research（资讯通信研究所）
Deep learning based methods have penetrated many image processing problems and become dominant solutions to these problems. A natural question raised here is "Is there any space for conventional methods on these problems?" In this paper, exposure interpolation is taken as an example to answer this question and the answer is "Yes". A framework on fusing conventional and deep learning method is introduced to generate an medium exposure image for two large-exposureratio images. Experimental results indicate that the quality of the medium exposure image is increased significantly through using the…
Tsinghua University（清华大学）；Anzhen Hospital, Capital Medical University（首都医科大学附属安贞医院）；Capital Medical University（首都医科大学）；The First Affiliated Hospital of Zhengzhou University（郑州大学第一附属医院）
Cardiac coronary angiography is a major technology to assist doctors during cardiac interventional surgeries. Under the exposure of X-ray radiation, doctors inject contrast agents through catheters to determine the position and status of coronary vessels in real time. To get a coronary angiography video with a high frame rate, the doctor needs to increase the exposure frequency and intensity of the X-ray. This will inevitably increase the X-ray harm to both patients and surgeons. In this work, we innovatively utilize a deep-learning based video interpolation algorithm to interpolate coronary a…
MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture
Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Baidu（百度）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Learning Event-guided Exposure-agnostic Video Frame Interpolation via Adaptive Feature Blending
School of Computing, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea（计算机学院，韩国科学技术院（KAIST），大田，韩国）；School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore（电子与电气工程学院，南洋理工大学（NTU），新加坡）
Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Louisiana State University（路易斯安那州立大学）
Through pixel-wise optical coding of images during exposure time, it is possible to extract sub-exposure images from a single capture. Such a capability can be used for different purposes, including high-speed imaging, high-dynamic-range imaging and compressed sensing. In this paper, we demonstrate a sub-exposure image extraction method, where the exposure coding pattern is inspired from frequency division multiplexing idea of communication systems. The coding masks modulate sub-exposure images in such a way that they are placed in non-overlapping regions in Fourier domain. The sub-exposure im…
Chandigarh Engineering College（昌迪加尔工程学院）；Thapar Institute of Engineering & Technology, University（塔帕尔工程技术大学）
Recent innovations shows that blending of details captured by single Low Dynamic Range (LDR) sensor overcomes the limitations of standard digital cameras to capture details from high dynamic range scene. We present a method to produce well-exposed fused image that can be displayed directly on conventional display devices. The ambition is to preserve details in poorly illuminated and brightly illuminated regions. Proposed approach does not require true radiance reconstruction and tone manipulation steps. The aforesaid objective is achieved by taking into account local information measure that s…
EVDI++: Event-based Video Deblurring and Interpolation via Self-Supervised Learning
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；Peng Cheng Laboratory（鹏城实验室）；Computer Graphics Lab of ETH Zurich（苏黎世联邦理工学院计算机图形实验室）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Fusing a sequence of perfectly aligned images captured at various exposures, has shown great potential to approach High Dynamic Range (HDR) imaging by sensors with limited dynamic range. However, in the presence of large motion of scene objects or the camera, mis-alignment is almost inevitable and leads to the notorious ``ghost'' artifacts. Besides, factors such as the noise in the dark region or color saturation in the over-bright region may also fail to fill local image details to the HDR image. This paper provides a novel multi-exposure fusion model based on Swin Transformer. Particularly, …
Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos. To this end, we present a unified framework of event-based motion deblurring and frame interpolation for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction. Specifically, the mapping relation between blurry frames and sharp latent images is first predicted by a learnable double integral network, and a fusion network is …
Unsupervised Learning Based Multi-Scale Exposure Fusion
school of Information Science and Engineering, Wuhan University of Science and Technology（武汉科技大学信息科学与工程学院）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属机构））
Neural Block-wise Auto-exposure for Single-Shot High Dynamic Range Imaging
Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高级智能与计算研究所）；school of Information Science and Engineering, Wuhan University of Science and Technology（武汉科技大学信息科学与工程学院）；Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局先进再制造与技术中心）
Kuaishou Technology（快手科技）；The Hong Kong University of Science and Technology（香港科技大学）
Capitalizing on the rapid development of neural networks, recent video frame interpolation (VFI) methods have achieved notable improvements. However, they still fall short for real-world videos containing large motions. Complex deformation and/or occlusion caused by large motions make it an extremely difficult problem in video frame interpolation. In this paper, we propose a simple yet effective solution, H-VFI, to deal with large motions in video frame interpolation. H-VFI contributes a hierarchical video interpolation transformer (HVIT) to learn a deformable kernel in a coarse-to-fine strate…
IIIT Hyderabad（海得拉巴国际信息技术学院）；University of California, Los Angeles（加州大学洛杉矶分校）；Georgia Tech（佐治亚理工学院）；IIT Gandhinagar（甘地纳格尔印度理工学院）
Due to hardware constraints, standard off-the-shelf digital cameras suffers from low dynamic range (LDR) and low frame per second (FPS) outputs. Previous works in high dynamic range (HDR) video reconstruction uses sequence of alternating exposure LDR frames as input, and align the neighbouring frames using optical flow based networks. However, these methods often result in motion artifacts in challenging situations. This is because, the alternate exposure frames have to be exposure matched in order to apply alignment using optical flow. Hence, over-saturation and noise in the LDR frames result…
Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research（资讯通信研究院）
Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying …
Computationally removing the motion blur introduced by camera shake or object motion in a captured image remains a challenging task in computational photography. Deblurring methods are often limited by the fixed global exposure time of the image capture process. The post-processing algorithm either must deblur a longer exposure that contains relatively little noise or denoise a short exposure that intentionally removes the opportunity for blur at the cost of increased noise. We present a novel approach of leveraging spatially varying pixel exposures for motion deblurring using next-generation …
Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames
AI Thrust, HKUST(GZ)（香港科技大学（广州）人工智能方向）；school of software Shandong University（山东大学软件学院）；School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（新加坡南洋理工大学电子与电气工程学院）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Immersive displays such as VR headsets, AR glasses, Multiview displays, Free point televisions have emerged as a new class of display technologies in recent years, offering a better visual experience and viewer engagement as compared to conventional displays. With the evolution of 3D video and display technologies, the consumer market for High Dynamic Range (HDR) cameras and displays is quickly growing. The lack of appropriate experimental data is a critical hindrance for the development of primary research efforts in the field of 3D HDR video technology. Also, the unavailability of sufficient…
High Dynamic Range Video Reconstruction from Single-Exposure Raw Sequences
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang Provincial Key Laboratory of Low Altitude Ubiquitous Networking Technology, Hangzhou Dianzi University（浙江省低空泛在组网技术重点实验室，杭州电子科技大学）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Technology（北京理工大学）
Huawei Technologies（华为技术有限公司）；Univ. of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
Recently, video frame interpolation using a combination of frame- and event-based cameras has surpassed traditional image-based methods both in terms of performance and memory efficiency. However, current methods still suffer from (i) brittle image-level fusion of complementary interpolation results, that fails in the presence of artifacts in the fused image, (ii) potentially temporally inconsistent and inefficient motion estimation procedures, that run for every inserted frame and (iii) low contrast regions that do not trigger events, and thus cause events-only motion estimation to generate a…
ExpoMotion: A Large-Scale Benchmark and A Householder Projection Network for Multi-Exposure Fusion
Nankai International Advanced Research Institute (SHENZHEN·FUTIAN)（南开大学国际研究院（深圳·福田））；Pengcheng Laboratory（鹏城实验室）；College of Computer Science, Nankai University（南开大学计算机科学学院）；The Hong Kong Polytechnic University（香港理工大学）；OPPO Research Institute（OPPO研究院）
Recent High Dynamic Range (HDR) techniques extend the capabilities of current cameras where scenes with a wide range of illumination can not be accurately captured with a single low-dynamic-range (LDR) image. This is generally accomplished by capturing several LDR images with varying exposure values whose information is then incorporated into a merged HDR image. While such approaches work well for static scenes, dynamic scenes pose several challenges, mostly related to the difficulty of finding reliable pixel correspondences. Data-driven approaches tackle the problem by learning an end-to-end …
E-3DGS: Gaussian Splatting with Exposure and Motion Events
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University（浙江大学现代光学仪器国家重点实验室）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；The Chair of Robotics, AI, and Real-Time Systems, Technical University of Munich（慕尼黑工业大学机器人、人工智能与实时系统教席）
In this paper, we propose a deep snapshot high dynamic range (HDR) imaging framework that can effectively reconstruct an HDR image from the RAW data captured using a multi-exposure color filter array (ME-CFA), which consists of a mosaic pattern of RGB filters with different exposure levels. To effectively learn the HDR image reconstruction network, we introduce the idea of luminance normalization that simultaneously enables effective loss computation and input data normalization by considering relative local contrasts in the "normalized-by-luminance" HDR domain. This idea makes it possible to …
University of Electronic Science and Technology of China（电子科技大学）
LiDAR point cloud frame interpolation, which synthesizes the intermediate frame between the captured frames, has emerged as an important issue for many applications. Especially for reducing the amounts of point cloud transmission, it is by predicting the intermediate frame based on the reference frames to upsample data to high frame rate ones. However, due to high-dimensional and sparse characteristics of point clouds, it is more difficult to predict the intermediate frame for LiDAR point clouds than videos. In this paper, we propose a novel LiDAR point cloud frame interpolation method, which …
FCNet: A Convolutional Neural Network for Arbitrary-Length Exposure Estimation
Nankai University（南开大学）；Beihang University（北京航空航天大学）；vivo Mobile Communication Co. Ltd.（维沃移动通信有限公司）；Guangdong University of Petrochemical Technology（广东石油化工学院）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Université Laval（拉瓦尔大学）
Scene inference under low-light is a challenging problem due to severe noise in the captured images. One way to reduce noise is to use longer exposure during the capture. However, in the presence of motion (scene or camera motion), longer exposures lead to motion blur, resulting in loss of image information. This creates a trade-off between these two kinds of image degradations: motion blur (due to long exposure) vs. noise (due to short exposure), also referred as a dual image corruption pair in this paper. With the rise of cameras capable of capturing multiple exposures of the same scene simu…
In this paper, we propose an algorithm to interpolate between a pair of images of a dynamic scene. While in the past years significant progress in frame interpolation has been made, current approaches are not able to handle images with brightness and illumination changes, which are common even when the images are captured shortly apart. We propose to address this problem by taking advantage of the existing optical flow methods that are highly robust to the variations in the illumination. Specifically, using the bidirectional flows estimated using an existing pre-trained flow network, we predic…
We address the problem of exposure correction of dark, blurry and noisy images captured in low-light conditions in the wild. Classical image-denoising filters work well in the frequency space but are constrained by several factors such as the correct choice of thresholds, frequency estimates etc. On the other hand, traditional deep networks are trained end-to-end in the RGB space by formulating this task as an image-translation problem. However, that is done without any explicit constraints on the inherent noise of the dark images and thus produce noisy and blurry outputs. To this end we propo…
AdaptiveAE: An Adaptive Exposure Strategy for HDR Capturing in Dynamic Scenes
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室，计算机科学学院）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学视觉技术国家工程研究中心，计算机科学学院）
Dual camera systems have assisted in the proliferation of various applications, such as optical zoom, low-light imaging and High Dynamic Range (HDR) imaging. In this work, we explore an optimal method for capturing the scene HDR and disparity map using dual camera setups. Hasinoff et al. (2010) have developed a noise optimal framework for HDR capture from a single camera. We generalize this to the dual camera set-up for estimating both HDR and disparity map. It may seem that dual camera systems can capture HDR in a shorter time. However, disparity estimation is a necessary step, which requires…
Seoul National University（首尔国立大学）；ETH Zürich（苏黎世联邦理工学院）
Videos in the real-world contain various dynamics and motions that may look unnaturally discontinuous in time when the recordedframe rate is low. This paper reports the second AIM challenge on Video Temporal Super-Resolution (VTSR), a.k.a. frame interpolation, with a focus on the proposed solutions, results, and analysis. From low-frame-rate (15 fps) videos, the challenge participants are required to submit higher-frame-rate (30 and 60 fps) sequences by estimating temporally intermediate frames. To simulate realistic and challenging dynamics in the real-world, we employ the REDS_VTSR dataset d…
Reconstruction of high-quality HDR images is at the core of modern computational photography. Significant progress has been made with multi-frame HDR reconstruction methods, producing high-resolution, rich and accurate color reconstructions with high-frequency details. However, they are still prone to fail in dynamic or largely over-exposed scenes, where frame misalignment often results in visible ghosting artifacts. Recent approaches attempt to alleviate this by utilizing an event-based camera (EBC), which measures only binary changes of illuminations. Despite their desirable high temporal re…
Hiroshima City University（广岛市立大学）；Kyushu University（九州大学）
When dynamic objects are captured by a camera, motion blur inevitably occurs. Such a blur is sometimes considered as just a noise, however, it sometimes gives an important effect to add dynamism in the scene for photographs or videos. Unlike the similar effects, such as defocus blur, which is now easily controlled even by smartphones, motion blur is still uncontrollable and makes undesired effects on photographs. In this paper, an unified framework to add motion blur on per-object basis is proposed. In the method, multiple frames are captured without motion blur and they are accumulated to cre…
Indian Institute of Science, Bangalore, India（印度科学理工学院（班加罗尔））
Enhancing practical low light raw images is a difficult task due to severe noise and color distortions from short exposure time and limited illumination. Despite the success of existing Convolutional Neural Network (CNN) based methods, their performance is not adaptable to different camera domains. In addition, such methods also require large datasets with short-exposure and corresponding long-exposure ground truth raw images for each camera domain, which is tedious to compile. To address this issue, we present a novel few-shot domain adaptation method to utilize the existing source camera lab…
Tsinghua University（清华大学）；Huawei Technologies Co. Ltd（华为技术有限公司）；Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）
Recording fast motion in a high FPS (frame-per-second) requires expensive high-speed cameras. As an alternative, interpolating low-FPS videos from commodity cameras has attracted significant attention. If only low-FPS videos are available, motion assumptions (linear or quadratic) are necessary to infer intermediate frames, which fail to model complex motions. Event camera, a new camera with pixels producing events of brightness change at the temporal resolution of $μs$ $(10^{-6}$ second $)$, is a game-changing device to enable video interpolation at the presence of arbitrarily complex motion. …
Kagoshima University（鹿儿岛大学）；Hiroshima City University（广岛市立大学）；Kyushu University（九州大学）
One of the solutions of depth imaging of moving scene is to project a static pattern on the object and use just a single image for reconstruction. However, if the motion of the object is too fast with respect to the exposure time of the image sensor, patterns on the captured image are blurred and reconstruction fails. In this paper, we impose multiple projection patterns into each single captured image to realize temporal super resolution of the depth image sequences. With our method, multiple patterns are projected onto the object with higher fps than possible with a camera. In this case, the…
Capturing Stable HDR Videos Using a Dual-Camera System
School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）；Hangzhou Dianzi University（杭州电子科技大学）；Lishui Institute of Hangzhou Dianzi University（杭州电子科技大学丽水学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
