Removing haze from real-world images is challenging due to unpredictable weather conditions, resulting in the misalignment of hazy and clear image pairs. In this paper, we propose an innovative dehazing framework that operates under non-aligned supervision. This framework is grounded in the atmospheric scattering model, and consists of three interconnected networks: dehazing, airlight, and transmission networks. In particular, we explore a non-alignment scenario that a clear reference image, unaligned with the input hazy image, is utilized to supervise the dehazing network. To implement this, we present a multi-scale reference loss that compares the feature representations between the referred image and the dehazed output. Our scenario makes it easier to collect hazy/clear image pairs in real-world environments, even under conditions of misalignment and shift views. To showcase the effectiveness of our scenario, we have collected a new hazy dataset including 415 image pairs captured by mobile Phone in both rural and urban areas, called "Phone-Hazy". Furthermore, we introduce a self-attention network based on mean and variance for modeling real infinite airlight, using the dark channel prior as positional guidance. Additionally, a channel attention network is employed to estimate the three-channel transmission. Experimental results demonstrate the superior performance of our framework over existing state-of-the-art techniques in the real-world image dehazing task. Phone-Hazy and code will be available at https://fanjunkai1.github.io/projectpage/NSDNet/index.html.
Image dehazing without paired haze-free images is of immense importance, as acquiring paired images often entails significant cost. However, we observe that previous unpaired image dehazing approaches tend to suffer from performance degradation near depth borders, where depth tends to vary abruptly. Hence, we propose to anneal the depth border degradation in unpaired image dehazing with cyclic perceptual-depth supervision. Coupled with the dual-path feature re-using backbones of the generators and discriminators, our model achieves $\mathbf{20.36}$ Peak Signal-to-Noise Ratio (PSNR) on NYU Dept…
CLIP-Guided Data Augmentation for Night-Time Image Dehazing
Hangzhou Dianzi University（杭州电子科技大学）；Jinan University（济南大学）；Hefei University of Technology（合肥工业大学）；Wuhan University（武汉大学）；The University of Tokyo（东京大学）
While Transformer has achieved remarkable performance in various high-level vision tasks, it is still challenging to exploit the full potential of Transformer in image restoration. The crux lies in the limited depth of applying Transformer in the typical encoder-decoder framework for image restoration, resulting from heavy self-attention computation load and inefficient communications across different depth (scales) of layers. In this paper, we present a deep and effective Transformer-based network for image restoration, termed as U2-Former, which is able to employ Transformer as the core oper…
Haze removal in aerial images is a challenging problem due to considerable variation in spatial details and varying contrast. Changes in particulate matter density often lead to degradation in visibility. Therefore, several approaches utilize multi-spectral data as auxiliary information for haze removal. In this paper, we propose SkyGAN for haze removal in aerial images. SkyGAN consists of 1) a domain-aware hazy-to-hyperspectral (H2H) module, and 2) a conditional GAN (cGAN) based multi-cue image-to-image translation module (I2I) for dehazing. The proposed H2H module reconstructs several visual…
Caused by the difference of data distributions, intra-domain gap and inter-domain gap are widely present in image processing tasks. In the field of image dehazing, certain previous works have paid attention to the inter-domain gap between the synthetic domain and the real domain. However, those methods only establish the connection from the source domain to the target domain without taking into account the large distribution shift within the target domain (intra-domain gap). In this work, we propose a Two-Step Dehazing Network (TSDN) with an intra-domain adaptation and a constrained inter-doma…
Frequency Domain-Based Diffusion Model for Unpaired Image Dehazing
Xi’an Jiaotong University（西安交通大学）；Wuhan University（武汉大学）；Insta360（影石）；Nanjing University of Science and Technology（南京理工大学）；University of California, Merced（加利福尼亚大学默塞德分校）
DADFNet: Dual Attention and Dual Frequency-Guided Dehazing Network for Video-Empowered Intelligent Transportation
Wuhan University of Technology（武汉理工大学）；Nanyang Technological University（南洋理工大学）；Sony AI（索尼人工智能研究院）；Singapore University of Technology and Design（新加坡科技设计大学）；Guangdong University of Technology（广东工业大学）
In most studies on learning-based image super-resolution (SR), the paired training dataset is created by downscaling high-resolution (HR) images with a predetermined operation (e.g., bicubic). However, these methods fail to super-resolve real-world low-resolution (LR) images, for which the degradation process is much more complicated and unknown. In this paper, we propose an unpaired SR method using a generative adversarial network that does not require a paired/aligned training dataset. Our network consists of an unpaired kernel/noise correction network and a pseudo-paired SR network. The cor…
Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）；Pazhou Lab（琶洲实验室）
Learning RAW-to-sRGB mapping has drawn increasing attention in recent years, wherein an input raw image is trained to imitate the target sRGB image captured by another camera. However, the severe color inconsistency makes it very challenging to generate well-aligned training pairs of input raw and target sRGB images. While learning with inaccurately aligned supervision is prone to causing pixel shift and producing blurry results. In this paper, we circumvent such issue by presenting a joint learning model for image alignment and RAW-to-sRGB mapping. To diminish the effect of color inconsistenc…
This work addresses the problem of semantic foggy scene understanding (SFSU). Although extensive research has been performed on image dehazing and on semantic scene understanding with clear-weather images, little attention has been paid to SFSU. Due to the difficulty of collecting and annotating foggy images, we choose to generate synthetic fog on real images that depict clear-weather outdoor scenes, and then leverage these partially synthetic data for SFSU by employing state-of-the-art convolutional neural networks (CNN). In particular, a complete pipeline to add synthetic fog to real, clear-…
Zhejiang University（浙江大学）；Westlake University（西lake大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hunan University（湖南大学）；Zhejiang University Shanghai Innovation Institute（浙江大学上海创新研究院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Simon Fraser University（西蒙弗雷泽大学）；IIT Kanpur（印度理工学院坎普尔分校）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions. A challenge in this task is how to handle 'off the path' scenarios where an agent veers from a reference path. Prior work supervises the agent with actions based on the shortest path from the agent's location to the goal, but such goal-oriented supervision is often not in alignment with the instruction. Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow. In this work, we prop…
University of Science Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Person re-identification (reID) aims to match person images to retrieve the ones with the same identity. This is a challenging task, as the images to be matched are generally semantically misaligned due to the diversity of human poses and capture viewpoints, incompleteness of the visible bodies (due to occlusion), etc. In this paper, we propose a framework that drives the reID network to learn semantics-aligned feature representation through delicate supervision designs. Specifically, we build a Semantics Aligning Network (SAN) which consists of a base network as encoder (SA-Enc) for re-ID, an…
Georgia Institute of Technology（佐治亚理工学院）；Microsoft Research（微软研究院）
Image alignment tasks require accurate pixel correspondences, which are usually recovered by matching local feature descriptors. Such descriptors are often derived using supervised learning on existing datasets with ground truth correspondences. However, the cost of creating such datasets is usually prohibitive. In this paper, we propose a new approach to align two images related by an unknown 2D homography where the local descriptor is learned from scratch from the images and the homography is estimated simultaneously. Our key insight is that a siamese convolutional neural network can be trai…
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC)（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE)（医学影像、机器人、分析计算与学习中心）；Suzhou Institute for Advanced Research, USTC（苏州先进研究院，中国科学技术大学）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；T Magnetic Resonance Translational Medicine Research Center, Department of Radiology, The First Affiliated Hospital (Southwest Hospital) of Army Medical University（7T磁共振转化医学研究中心，放射科，中国医学大学第一附属医院（西南医院））
Weakly-supervised vision-language (V-L) pre-training (W-VLP) aims at learning cross-modal alignment with little or no paired data, such as aligned images and captions. Recent W-VLP methods, which pair visual features with object tags, help achieve performances comparable with some VLP models trained with aligned pairs in various V-L downstream tasks. This, however, is not the case in cross-modal retrieval (XMR). We argue that the learning of such a W-VLP model is curbed and biased by the object tags of limited semantics. We address the lack of paired V-L data for model supervision with a novel…
Standard practice in pretraining multimodal models, such as vision-language models, is to rely on pairs of aligned inputs from both modalities, for example, aligned image-text pairs. However, such pairs can be difficult to obtain in low-resource settings and for some modality pairs (e.g., structured tables and images). In this work, we investigate the extent to which we can reduce the reliance on such parallel data, which we term \emph{bimodal supervision}, and use models that are pretrained on each modality independently. We experiment with a high-performing vision-language model, and analyze…
Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery
Beijing University of Technology（北京工业大学）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing Institute of Technology（北京理工大学）；Fuzhou University（福州大学）；China University of Geosciences Wuhan（中国地质大学（武汉））；Ghent University（根特大学）
Cascaded Interaction with Eroded Deep Supervision for Salient Object Detection
Institute of Space Science and Applied Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）空间科学与应用技术研究院）；School of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）机电工程与自动化学院）
In this paper, we propose a method for keypoint discovery from a 2D image using image-level supervision. Recent works on unsupervised keypoint discovery reliably discover keypoints of aligned instances. However, when the target instances have high viewpoint or appearance variation, the discovered keypoints do not match the semantic correspondences over different images. Our work aims to discover keypoints even when the target instances have high viewpoint and appearance variation by using image-level supervision. Motivated by the weakly-supervised learning approach, our method exploits image-l…
University of Maryland, Baltimore County（马里兰大学巴尔的摩郡分校）；Facebook AI（Facebook 人工智能研究院）；UC Davis（加州大学戴维斯分校）
Using natural language as a supervision for training visual recognition models holds great promise. Recent works have shown that if such supervision is used in the form of alignment between images and captions in large training datasets, then the resulting aligned models perform well on zero-shot classification as downstream tasks2. In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models. Through extensive and careful experiments, we show that: 1) A simple Bag-of-Words (BoW) caption could be used as a replac…
Learning What Helps: Task-Aligned Context Selection for Vision Tasks
KTH Royal Institute of Technology（皇家理工学院）；Science for Life Laboratory（生命科学实验室）；Karolinska Institutet（卡罗林斯卡研究院）；Pathology, Clinical Pharmacology and Safety Sciences, R&D AstraZeneca（病理学、临床药理学与安全性科学，阿斯利康研发部）
Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
Graduate School of Arts and Sciences, The University of Tokyo（东京大学文学部与科学研究科）；Graduate School of Engineering Science, The University of Osaka（大阪大学工程科学研究科）
Example-guided image synthesis has been recently attempted to synthesize an image from a semantic label map and an exemplary image. In the task, the additional exemplary image serves to provide style guidance that controls the appearance of the synthesized output. Despite the controllability advantage, the previous models are designed on datasets with specific and roughly aligned objects. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically unaligned to the given label map. To this end, we first propose a new Masked S…
Principle-Guided Supervision for Interpretable Uncertainty in Medical Image Segmentation
School of Data Science, Fudan University（复旦大学数据科学学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑启发智能科学研究院）；National Heart and Lung Institute, Imperial College London（伦敦帝国理工学院国家心脏和肺研究所）
Hong Kong University of Science and Technology（香港科技大学）；Beihang University（北京航空航天大学）
Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground…
Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In the task, the additional exemplar image provides the style guidance that controls the appearance of the synthesized output. Despite the controllability advantage, the existing models are designed on datasets with specific and roughly aligned objects. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. To this end, we first propose a Masked Spatial-C…
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；JD Explore Academy at JD.com（京东集团京东探索研究院）
Text-to-image person re-identification (ReID) aims to search for images containing a person of interest using textual descriptions. However, due to the significant modality gap and the large intra-class variance in textual descriptions, text-to-image ReID remains a challenging problem. Accordingly, in this paper, we propose a Semantically Self-Aligned Network (SSAN) to handle the above problems. First, we propose a novel method that automatically extracts semantically aligned part-level features from the two modalities. Second, we design a multi-view non-local network that captures the relatio…
The Hebrew University（希伯来大学）；Tel-Aviv University（特拉维夫大学）；Adobe Research（Adobe研究院）
In this paper, we perform an in-depth study of the properties and applications of aligned generative models. We refer to two models as aligned if they share the same architecture, and one of them (the child) is obtained from the other (the parent) via fine-tuning to another domain, a common practice in transfer learning. Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation. Here, we perform the first detailed exploration of model alignment, also focusing on StyleGAN. First, we empirically analyze aligned models and provide answers…
Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local …
