用于零样本高光谱 pansharpening 的双模态提示扩散先验

中文摘要 AI 辅助

高光谱 pansharpening 旨在从全色（PAN）图像和低分辨率高光谱（LRHS）图像重建高分辨率高光谱（HRHS）图像，同时保留空间细节和光谱保真度。近期基于扩散的方法通过生成低维表示并将其映射至高光谱域来利用预训练图像先验。然而，观测到的全色图像和高光谱图像通常仅通过外部重建目标施加，限制了它们与扩散先验的直接交互。为解决该问题，我们提出用于零样本高光谱 pansharpening 的双模态图像提示扩散模型（DIDM）。DIDM 将低分辨率高光谱和全色观测分别编码为光谱和空间提示令牌，并通过交叉注意力将其注入冻结遥感扩散模型的中间特征，使互补的光谱和空间信息能直接引导扩散特征演化。此外，我们引入全色引导的加权像素感知总变分正则化器，其结合低分辨率高光谱退化保真度和全色响应保真度，并带有梯度自适应结构正则化，从而保留结构不连续性，同时抑制同质区域的虚假变化。在降分辨率协议下对 Pavia、Chikusei 和 Houston 进行的大量实验表明，DIDM 在所有评估指标中均取得最佳性能；而在 FR1 上的全分辨率评估中，其获得了对比方法中最高的 HQNR。这些结果证明，内部双模态提示和全色引导结构正则化在空间细节增强与光谱保留之间提供了有效平衡。

英文摘要

Hyperspectral pansharpening aims to reconstruct a high resolution hyperspectral (HRHS) image from a panchromatic (PAN) image and a low resolution hyperspectral (LRHS) image while preserving both spatial details and spectral fidelity. Recent diffusion based methods exploit pretrained image priors by generating a low dimensional representation and subsequently mapping it to the hyperspectral domain. However, the observed panchromatic and hyperspectral images are typically imposed only through external reconstruction objectives, limiting their direct interaction with the diffusion prior. To address this issue, we propose dual-modality image-prompted diffusion model (DIDM) for zero shot hyperspectral pansharpening. DIDM encodes the low resolution hyperspectral and panchromatic observations into spectral and spatial prompt tokens, respectively, and injects them into intermediate features of a frozen remote sensing diffusion model through cross attention, allowing complementary spectral and spatial information to directly guide diffusion feature evolution. In addition, we introduce a panchromatic guided weighted pixel aware total variation regularizer that combines low resolution hyperspectral degradation fidelity and panchromatic response fidelity with gradient adaptive structural regularization, thereby preserving structural discontinuities while suppressing spurious variations in homogeneous regions. Extensive experiments on Pavia, Chikusei, and Houston under reduced resolution protocols show that DIDM achieves the best performance across all evaluated metrics, while full resolution evaluation on FR1 yields the highest HQNR among the compared methods. These results demonstrate that internal dual modality prompting and panchromatic guided structural regularization provide an effective balance between spatial detail enhancement and spectral preservation.

发表机构

• School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
• Peking University（北京大学）
• National Center for Applied Mathematics Shenzhen (NCAMS), Southern University of Science and Technology（南方科技大学深圳应用数学国家中心）
• Department of Automation, Tsinghua University（清华大学自动化系）
• National Research Council of Italy, Institute of Integrated Methodologies for Earth Observation (CNR-IMIOT)（意大利国家研究委员会综合地球观测方法研究所）

机构由 AI 辅助整理，请以论文原文为准。