SMARTIES：用于遥感图像的光谱感知多传感器自编码器
SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images
AI总结：
SMARTIES是一个光谱感知的多传感器基础模型，通过共享空间投影和跨传感器令牌混合实现任意波段组合训练，提升多传感器遥感任务的性能与泛化。
摘要：
从光学传感器到微波雷达，利用遥感（RS）传感器的互补优势对于实现对地球的密集时空监测至关重要。相比之下，最近的深度学习模型，无论是任务特定的还是基础性的，通常都针对单一传感器或固定组合：将这些模型适配到不同的传感输入需要架构更改和重新训练，限制了跨多个RS传感器的可扩展性和泛化能力。相反，一个能够调节其特征表示以接受多种传感器输入的单一模型，将为敏捷灵活的多传感器RS数据处理铺平道路。为解决这一问题，我们引入了SMARTIES，一个通用且多用途的基础模型，它消除了传感器特定/依赖的工作，并实现了对多样化RS传感器的可扩展性和泛化：SMARTIES将来自异构传感器的数据投影到一个共享的光谱感知空间中，从而允许在训练和推理中使用任意波段组合。为了获得与传感器无关的表示，我们训练了一个统一的Transformer模型，通过跨传感器令牌混合来重建掩码的多传感器数据。在跨多种传感器的单模态和多模态任务上，SMARTIES优于依赖传感器特定预训练的先前模型。我们的代码和预训练模型可在https://gsumbul.github.io/SMARTIES获取。
英文摘要：
From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet. In contrast, recent deep learning models, whether task-specific or foundational, are often specific to single sensors or to fixed combinations: adapting such models to different sensory inputs requires both architectural changes and re-training, limiting scalability and generalization across multiple RS sensors. On the contrary, a single model able to modulate its feature representations to accept diverse sensors as input would pave the way to agile and flexible multi-sensor RS data processing. To address this, we introduce SMARTIES, a generic and versatile foundation model lifting sensor-specific/dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, we train a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup. On both single- and multi-modal tasks across diverse sensors, SMARTIES outperforms previous models that rely on sensor-specific pretraining. Our code and pretrained models are available at https://gsumbul.github.io/SMARTIES.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
