There exists an intrinsic relationship between the spectral sensitivity of the Human Visual System (HVS) and colour perception; these intertwined phenomena are often overlooked in perceptual compression research. In general, most previously proposed visually lossless compression techniques exploit luminance (luma) masking including luma spatiotemporal masking, luma contrast masking and luma texture/edge masking. The perceptual relevance of color in a picture is often overlooked, which constitutes a gap in the literature. With regard to the spectral sensitivity phenomenon of the HVS, the color channels of raw RGB 4:4:4 data contain significant color-based psychovisual redundancies. These perceptual redundancies can be quantized via color channel-level perceptual quantization. In this paper, we propose a novel spatiotemporal visually lossless coding method named Spectral Perceptual Quantization (Spectral-PQ). With application for RGB 4:4:4 video data, Spectral-PQ exploits HVS spectral sensitivity-related color masking in addition to spatial masking and temporal masking; the proposed method operates at the Coding Block (CB) level and the Prediction Unit (PU) level in the HEVC standard. Spectral-PQ perceptually adjusts the Quantization Step Size (QStep) at the CB level if high variance spatial data in G, B and R CBs is detected and also if high motion vector magnitudes in PUs are detected. Compared with anchor 1 (HEVC HM 16.17 RExt), Spectral-PQ considerably reduces bitrates with a maximum reduction of approximately 81%. The Mean Opinion Score (MOS) in the subjective evaluations show that Spectral-PQ successfully achieves perceptually lossless quality.
The success of image perturbations that are designed to fool image classifier is assessed in terms of both adversarial effect and visual imperceptibility. The conventional assumption on imperceptibility is that perturbations should strive for tight $L_p$-norm bounds in RGB space. In this work, we drop this assumption by pursuing an approach that exploits human color perception, and more specifically, minimizing perturbation size with respect to perceptual color distance. Our first approach, Perceptual Color distance C&W (PerC-C&W), extends the widely-used C&W approach and produces larger RGB p…
University of Innsbruck（因斯布鲁克大学）；LEC GmbH（LEC有限公司）；INNIO Jenbacher GmbH & Co OG（INNIO颜巴赫两合公司）
Real-time estimation of actual object depth is an essential module for various autonomous system tasks such as 3D reconstruction, scene understanding and condition assessment. During the last decade of machine learning, extensive deployment of deep learning methods to computer vision tasks has yielded approaches that succeed in achieving realistic depth synthesis out of a simple RGB modality. Most of these models are based on paired RGB-depth data and/or the availability of video sequences and stereo images. The lack of sequences, stereo data and RGB-depth pairs makes depth estimation a fully …
MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；StepFun（阶跃星辰）；Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information（多模态信息超级智能安全北京市重点实验室）；School of Information Science and Technology, Shanghai Tech University（上海科技大学信息科学与技术学院）
Technical University of Munich（慕尼黑工业大学）；Google（谷歌）
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervised approach learns to jointly inpaint geometry and color by correlating an incomplete RGB-D scan with a more complete version of that scan. Notably, rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored …
TUBITAK Space Technologies Research Institute（土耳其科学技术研究理事会空间技术研究所）
Cloud detection from remotely observed data is a critical pre-processing step for various remote sensing applications. In particular, this problem becomes even harder for RGB color images, since there is no distinct spectral pattern for clouds, which is directly separable from the Earth surface. In this paper, we adapt a deep pyramid network (DPN) to tackle this problem. For this purpose, the network is enhanced with a pre-trained parameter model at the encoder layer. Moreover, the method is able to obtain accurate pixel-level segmentation and classification results from a set of noisy labeled…
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Can faces acquired by low-cost depth sensors be useful to catch some characteristic details of the face? Typically the answer is no. However, new deep architectures can generate RGB images from data acquired in a different modality, such as depth data. In this paper, we propose a new \textit{Deterministic Conditional GAN}, trained on annotated RGB-D face datasets, effective for a face-to-face translation from depth to RGB. Although the network cannot reconstruct the exact somatic features for unknown individual faces, it is capable to reconstruct plausible faces; their appearance is accurate e…
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Event cameras are biologically-inspired sensors that gather the temporal evolution of the scene. They capture pixel-wise brightness variations and output a corresponding stream of asynchronous events. Despite having multiple advantages with respect to traditional cameras, their use is partially prevented by the limited applicability of traditional data processing and vision algorithms. To this aim, we present a framework which exploits the output stream of event cameras to synthesize RGB frames, relying on an initial or a periodic set of color key-frames and the sequence of intermediate events…
Convolutional Neural Networks (CNNs) trained on large scale RGB databases have become the secret sauce in the majority of recent approaches for object categorization from RGB-D data. Thanks to colorization techniques, these methods exploit the filters learned from 2D images to extract meaningful representations in 2.5D. Still, the perceptual signature of these two kind of images is very different, with the first usually strongly characterized by textures, and the second mostly by silhouettes of objects. Ideally, one would like to have two CNNs, one for RGB and one for depth, each trained on a …
Rethinking RGB Color Representation for Image Restoration Models
Seoul National University（首尔大学）；Hanyang University（汉阳大学）；ASRI（ASRI（高级科学技术研究所））；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能跨学科项目）
In perceptual image coding applications, the main objective is to decrease, as much as possible, Bits Per Pixel (BPP) while avoiding noticeable distortions in the reconstructed image. In this paper, we propose a novel perceptual image coding technique, named Perceptual Color Compression (PCC). PCC is based on a novel model related to Human Visual System (HVS) spectral sensitivity and CIELAB Just Noticeable Color Difference (JNCD). We utilize this modeling to capitalize on the inability of the HVS to perceptually differentiate photons in very similar wavelength bands (e.g., distinguishing very …
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Higher School of Economics（高等经济学院）；New York University（纽约大学）
RGBD images, combining high-resolution color and lower-resolution depth from various types of depth sensors, are increasingly common. One can significantly improve the resolution of depth maps by taking advantage of color information; deep learning methods make combining color and depth information particularly easy. However, fusing these two sources of data may lead to a variety of artifacts. If depth maps are used to reconstruct 3D shapes, e.g., for virtual reality applications, the visual quality of upsampled images is particularly important. The main idea of our approach is to measure the …
National Technical University of Athens（雅典国立技术大学）；Imperial College London（伦敦帝国理工学院）；Institute for Language and Speech Processing, Athena R.C.（雅典娜研究中心语言与语音处理研究所）；Foundation for Research & Technology - Hellas（希腊研究与技术基金会）；University of Exeter（埃克塞特大学）
The recent state of the art on monocular 3D face reconstruction from image data has made some impressive advancements, thanks to the advent of Deep Learning. However, it has mostly focused on input coming from a single RGB image, overlooking the following important factors: a) Nowadays, the vast majority of facial image data of interest do not originate from single images but rather from videos, which contain rich dynamic information. b) Furthermore, these videos typically capture individuals in some form of verbal communication (public talks, teleconferences, audiovisual human-computer intera…
Recent promising effort for spectral reconstruction (SR) focuses on learning a complicated mapping through using a deeper and wider convolutional neural networks (CNNs). Nevertheless, most CNN-based SR algorithms neglect to explore the camera spectral sensitivity (CSS) prior and interdependencies among intermediate features, thus limiting the representation ability of the network and performance of SR. To conquer these issues, we propose a novel adaptive weighted attention network (AWAN) for SR, whose backbone is stacked with multiple dual residual attention blocks (DRAB) decorating with long …
A Conditional U-Net Pipeline with Pre- and Post-Processing for Aerial RGB-to-Thermal Image Translation
Department of Data Science, University of Michigan, Ann Arbor, MI, USA（数据科学系，密歇根大学，安阿伯，MI，美国）；Department of Information Science, University of Michigan, Ann Arbor, MI, USA（信息科学系，密歇根大学，安阿伯，MI，美国）；Department of Computer Science, University of Michigan, Ann Arbor, MI, USA（计算机科学系，密歇根大学，安阿伯，MI，美国）；Arcknow, New York, USA（Arcknow，纽约，美国）；School of Environmental Sustainability, University of Michigan, Ann Arbor, MI, USA（可持续环境学院，密歇根大学，安阿伯，MI，美国）；SmithGroup, Ann Arbor, MI, USA（SmithGroup，安阿伯，MI，美国）；Michigan Institute for Data and AI in Society (MIDAS), University of Michigan, Ann Arbor, MI, USA（密歇根数据与人工智能社会研究院（MIDAS），密歇根大学，安阿伯，MI，美国）；United States International University (USIU), Nairobi, Kenya（美国国际大学（USIU），内罗毕，肯尼亚）；Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA（学习健康科学系，密歇根大学医学院，安阿伯，MI，美国）；Department of Pharmacology, University of Michigan Medical School, Ann Arbor, MI, USA（药理学系，密歇根大学医学院，安阿伯，MI，美国）；Center for Global Health Equity, University of Michigan, Ann Arbor, MI, USA（全球健康公平中心，密歇根大学，安阿伯，MI，美国）
CSNR and JMIM Based Spectral Band Selection for Reducing Metamerism in Urban Driving
School of Engineering, University of Galway, Ireland（Galway大学工程学院，爱尔兰）；Ryan Institute, University of Galway, Ireland（Galway大学Ryan学院，爱尔兰）；Valeo Vision Systems, Tuam, Ireland（爱尔兰Tuam市Valeo Vision Systems公司）
Learned Lightweight Smartphone ISP with Unpaired Data
Faculty of Computer Science Alexandru Ioan Cuza University of Iasi（亚历山德鲁·伊奥安·库扎大学计算机科学系）；Computer Vision Lab, CAIDAS & IFI University of Würzburg（弗赖堡大学计算机视觉实验室）
