探索大规模预训练模型在恶劣天气去除中的应用
Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal
AI总结：
本研究探索大规模预训练模型在恶劣天气图像复原中的应用，提出SAR编码器、CLIP-SRD蒸馏策略及CWP嵌入模块，在多种恶劣天气下实现了当前最优性能。
摘要：
恶劣天气条件（如雨、雪和雾）下的图像复原是计算机视觉的一个基础问题，对各类下游应用具有重要意义。与早期针对特定天气类型专门设计的方法不同，近期大多数研究倾向于通过空间特征表示学习或语义信息嵌入同时去除多种恶劣天气影响。受大规模预训练模型（如CLIP）各类成功应用的启发，本文从空间特征表示学习和语义信息嵌入两方面探索其对该任务的潜在增益：1）在空间特征表示学习方面，我们设计了空间自适应残差（Spatially-Adaptive Residual，SAR）编码器，用于自适应提取退化区域。为促进其训练，我们提出软残差蒸馏（Soft Residual Distillation，CLIP-SRD）策略，从CLIP中迁移干净图像与恶劣天气图像间的空间知识；2）在语义信息嵌入方面，我们提出CLIP天气先验（CLIP Weather Prior，CWP）嵌入模块，使网络能自适应处理不同天气条件。该模块将CLIP图像编码器提取的样本特定天气先验与一组参数学习到的分布特定信息相整合，并通过交叉注意力机制完成嵌入。大量实验表明，本文提出的方法在不同且具有挑战性的恶劣天气条件下均能达到最优性能。代码将公开。
英文摘要：
Image restoration under adverse weather conditions (e.g., rain, snow and haze) is a fundamental computer vision problem and has important indications for various downstream applications. Different from early methods that are specially designed for specific type of weather, most recent works tend to remove various adverse weather effects simultaneously through either spatial feature representation learning or semantic information embedding. Inspired by the various successful applications of large-scale pre-trained models (e.g, CLIP), in this paper, we explore the potential benefits of them for this task through both spatial feature representation learning and semantic information embedding aspects: 1) for spatial feature representation learning, we design a Spatially-Adaptive Residual (\textbf{SAR}) Encoder to extract degraded areas adaptively. To facilitate its training, we propose a Soft Residual Distillation (\textbf{CLIP-SRD}) strategy to transfer the spatial knowledge from CLIP between clean and adverse weather images; 2) for semantic information embedding, we propose a CLIP Weather Prior (\textbf{CWP}) embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters, and embeds them through a cross attention mechanism. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art performance under different and challenging adverse weather conditions. Code will be made available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
