arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.05808cs.CVeess.IV
收藏

面向盲图像超分辨率的空间变化核细化与扩散模型的自适应多模态融合

Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with Diffusion Model for Blind Image Super-Resolution

Shanghai Engineering Research Center of AI & Robotics(上海市AI与机器人工程研究中心);Academy for Engineering & Technology, Fudan University(复旦大学工程技术研究院);East China University of Science and Technology(华东理工大学);College of Information Technology, Shanghai Ocean University(上海海洋大学信息技术学院)

Junxiong Lin, Yan Wang, Zeng Tao, Boyang Wang, Qing Zhao, Haorang Wang, Xuan Tong, Xinji Mai, Yuxuan Lin, Wei Song, Jiawen Yu, Shaoqi Yan, Wenqiang Zhang

AI总结:

针对现有扩散式盲图像超分辨率方法忽略退化约束、未考虑模糊核空间可变性的问题,提出SSR框架,通过SVKR模块实现深度图与空间变化模糊核的互增强,再经AMF模块对齐多模态信息约束扩散过程,生成更真实的超分辨率结果。

摘要:

用于图像生成的预训练扩散模型蕴含着大量与复杂纹理相关的先验知识。挖掘这些先验知识在图像超分辨率任务中的潜力是一个极具吸引力的研究方向。然而,当前主流的基于扩散的方法忽略了退化信息对扩散过程的约束,且未考虑开放环境中由运动抖动、失焦等因素导致的估计模糊核固有的空间可变性,这一疏漏使得图像超分辨率效果与实际情况存在显著偏差。针对上述问题,我们提出了名为SSR(空间变化核细化与扩散模型自适应多模态融合盲图像超分辨率)的框架。在SSR框架中,我们设计了空间变化核细化(SVKR)模块:该模块估计一种融合深度信息的深度感知核(Depth-Informed Kernel),该核具有空间变化特性;同时SVKR可提升从低分辨率(LR)图像中获取的深度信息的精度,实现深度图与模糊核估计的相互增强。最后,我们引入自适应多模态融合(AMF)模块,对低分辨率图像、深度图、模糊核三种模态的信息进行对齐,以此约束扩散模型生成更真实的超分辨率(SR)结果。

英文摘要:

Pre-trained diffusion models utilized for image generation encapsulate a substantial reservoir of a priori knowledge pertaining to intricate textures. Harnessing the potential of leveraging this a priori knowledge in the context of image super-resolution presents a compelling avenue. Nonetheless, prevailing diffusion-based methodologies presently overlook the constraints imposed by degradation information on the diffusion process. Furthermore, these methods fail to consider the spatial variability inherent in the estimated blur kernel, stemming from factors such as motion jitter and out-of-focus elements in open-environment scenarios. This oversight results in a notable deviation of the image super-resolution effect from fundamental realities. To address these concerns, we introduce a framework known as Adaptive Multi-modal Fusion of \textbf{S}patially Variant Kernel Refinement with Diffusion Model for Blind Image \textbf{S}uper-\textbf{R}esolution (SSR). Within the SSR framework, we propose a Spatially Variant Kernel Refinement (SVKR) module. SVKR estimates a Depth-Informed Kernel, which takes the depth information into account and is spatially variant. Additionally, SVKR enhance the accuracy of depth information acquired from LR images, allowing for mutual enhancement between the depth map and blur kernel estimates. Finally, we introduce the Adaptive Multi-Modal Fusion (AMF) module to align the information from three modalities: low-resolution images, depth maps, and blur kernels. This alignment can constrain the diffusion model to generate more authentic SR results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑