基于Perceiver IO融合卫星和街景图像的多模态建筑检测
Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery
AI总结：
提出一种通过Perceiver IO架构融合卫星和街景图像的多模态分类框架，使用共享DINOv2骨干网络的空间补丁令牌，无需填充或固定大小池化即可处理可变数量的街景视图，并联合预测屋顶元素和材料类别，在包含10个国家32135栋建筑的数据集上验证了RGB-M掩码策略和融合模型的有效性。
摘要：
我们提出了一种多模态分类框架，通过Perceiver IO架构融合卫星和街景图像，该架构基于共享DINOv2骨干网络的空间补丁令牌。该设计自然地处理每栋建筑可变数量的街景视图，无需填充或固定大小池化，并联合预测多标签屋顶元素和屋顶材料类别。我们构建了一个包含10个国家32,135栋建筑（61,672个片段）的大规模数据集，将卫星图像与每个片段最多八个街景视图配对，并评估了四种用于隔离目标建筑的掩码策略。我们提出了一种RGB-M掩码策略，将建筑足迹掩码作为第四个输入通道，提供了一种软空间先验，在两种模态下均优于硬裁剪。Perceiver IO融合模型优于所有其他融合策略，并在街景可见的属性上取得了显著的每类增益（例如，石板+11.3 AP，老虎窗+1.3 AP），尽管仅卫星基线在主要从上方可见的类别的宏观平均mAP上仍保持轻微优势。这些结果为多模态建筑检测建立了一种可扩展、灵活的架构，能够处理异构输入和多个输出任务。
英文摘要：
We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture operating on spatial patch tokens from a shared DINOv2 backbone. The design naturally handles a variable number of street-level views per building without padding or fixed-size pooling, and jointly predicts multi-label roof element and roof material classes. We construct a large-scale dataset of 32,135 buildings (61,672 segments) spanning ten countries, pairing satellite images with up to eight street-level views per segment and evaluating four masking strategies for isolating the target building. We propose an RGB-M masking strategy that appends the building footprint mask as a fourth input channel, providing a soft spatial prior that outperforms hard cropping across both modalities. The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level (e.g., +11.3 AP for slate, +1.3 AP for dormers), though the satellite-only baseline retains a slight advantage in macro-averaged mAP for classes that are predominantly visible from above. These results establish a scalable, flexible architecture for multi-modal building inspection that can accommodate heterogeneous inputs and multiple output tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
