CROMA：基于对比雷达-光学掩码自编码器的遥感表示
CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders
AI总结：
CROMA通过结合对比学习与掩码重建的自监督框架，并引入X-/2D-ALiBi空间偏置策略，在遥感多模态数据上学习到优于现有最优模型的单模态与多模态表示，在多个分类与分割基准上取得显著提升。
摘要：
遥感作为一项至关重要且快速发展的应用，提供了海量但标签稀疏、空间对齐的多模态数据；这使得自监督学习算法变得不可或缺。我们提出了CROMA：一个结合对比与重建自监督目标的框架，用于学习丰富的单模态和多模态表示。我们的方法分别对在空间和时间上对齐的、被掩码的多光谱光学样本和合成孔径雷达样本进行编码，并执行跨模态对比学习。另一个编码器融合这些传感器，生成联合多模态编码，并通过一个轻量级解码器用于预测被掩码的图像块。我们表明，当利用空间对齐的多模态数据时，这些目标是互补的。我们还引入了X-ALiBi和2D-ALiBi，它们对我们的交叉注意力和自注意力矩阵施加空间偏置。这些策略改进了表示，并使我们的模型能够在测试时有效外推到最大17.6倍大的图像。CROMA优于当前最先进的多光谱模型，在以下评估中表现更佳：四个分类基准——微调（平均提升1.8%）、线性探测（平均提升2.4%）和非线性探测（平均提升1.4%）、kNN分类（平均提升3.5%）以及K-means聚类（平均提升8.4%）；以及三个分割基准（平均提升6.4%）。CROMA丰富的、可选的多模态表示可以在遥感应用中得到广泛利用。
英文摘要：
A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks -- finetuning (avg. 1.8%), linear (avg. 2.4%) and nonlinear (avg. 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg. 6.4%). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
