BirdSAT：用于鸟类物种分类与制图的跨视角对比掩码自编码器
BirdSAT: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mapping
AI总结：
本文提出BirdSAT框架，融合对比学习与掩码图像建模并引入元数据，在跨视角鸟类数据集上训练ViT，实现细粒度分类与跨模态检索的SotA性能，并支持物种分布制图。
摘要：
我们提出了一种元数据感知的自监督学习（SSL）框架，可用于全球鸟类物种的细粒度分类和生态制图。该框架统一了两种自监督学习策略：对比学习（CL）和掩码图像建模（MIM），同时利用鸟类地面图像中可用的元数据丰富嵌入空间。我们在一个新颖的跨视角全球鸟类物种数据集上分别训练单模态和跨模态ViT，该数据集包含地面图像、元数据（位置、时间）以及相应的卫星图像。我们通过在两个下游任务——细粒度视觉分类（FGVC）和跨模态检索——上的评估，证明我们的模型学习到了鸟类的细粒度和地理条件化特征。使用我们框架学习的预训练模型在iNAT-2021鸟类的FGVC上以及CUB-200-2011和NABirds数据集的迁移学习设置中均取得了SotA性能。此外，我们模型出色的跨模态检索性能使得能够创建任意地理区域的物种分布图。数据集和源代码将在https://github.com/mvrl/BirdSAT发布。
英文摘要：
We propose a metadata-aware self-supervised learning~(SSL)~framework useful for fine-grained classification and ecological mapping of bird species around the world. Our framework unifies two SSL strategies: Contrastive Learning~(CL) and Masked Image Modeling~(MIM), while also enriching the embedding space with metadata available with ground-level imagery of birds. We separately train uni-modal and cross-modal ViT on a novel cross-view global bird species dataset containing ground-level imagery, metadata (location, time), and corresponding satellite imagery. We demonstrate that our models learn fine-grained and geographically conditioned features of birds, by evaluating on two downstream tasks: fine-grained visual classification~(FGVC) and cross-modal retrieval. Pre-trained models learned using our framework achieve SotA performance on FGVC of iNAT-2021 birds and in transfer learning settings for CUB-200-2011 and NABirds datasets. Moreover, the impressive cross-modal retrieval performance of our model enables the creation of species distribution maps across any geographic region. The dataset and source code will be released at https://github.com/mvrl/BirdSAT}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
