arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.14969cs.CV
收藏

基于预训练遥感Transformers的迁移学习

Transfer Learning with Pretrained Remote Sensing Transformers

Anthony Fuller, Koreen Millard, James R. Green

摘要:

虽然遥感界已经开始对Transformers进行预训练(旨在对遥感任务进行微调),但尚不清楚这些模型在分布偏移下的表现。在这里,我们在130万张卫星来源的RS图像上预训练一个新的RS Transformer(称为SatViT-V2),然后对其进行微调(以及其他五个模型),以研究它在训练期间未看到的分布上的表现。我们将一个专业标记的土地覆盖数据集分为14个基于源生物群系的数据集。我们分别在每个生物群落上对每个模型进行训练,并在所有其他生物群落上进行测试。总共进行了1638次生物群落转移实验。微调后,我们发现SatViT-V2在分布内(匹配生物群落)和分布外(不匹配生物群落)数据上分别比SatViT-V1高出3.1%和2.8%。此外,我们发现,从线性探测的解决方案(即,利用LPFT [1])将SatViT-V2在分发内数据和分发外数据上的性能分别提高了1.2%和2.4%。接下来,我们发现预训练的RS Transformers在分布偏移下比非预训练的模型更好地校准,并且利用LPFT可以进一步改进模型校准。最后,我们发现,五个措施的分布偏移适度相关的生物群落转移性能。我们共享代码和预训练的模型权重。(https://github.com/antofuller/SatViT)

英文摘要:

Although the remote sensing (RS) community has begun to pretrain transformers (intended to be fine-tuned on RS tasks), it is unclear how these models perform under distribution shifts. Here, we pretrain a new RS transformer--called SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along with five other models) to investigate how it performs on distributions not seen during training. We split an expertly labeled land cover dataset into 14 datasets based on source biome. We train each model on each biome separately and test them on all other biomes. In all, this amounts to 1638 biome transfer experiments. After fine-tuning, we find that SatViT-V2 outperforms SatViT-V1 by 3.1% on in-distribution (matching biomes) and 2.8% on out-of-distribution (mismatching biomes) data. Additionally, we find that initializing fine-tuning from the linear probed solution (i.e., leveraging LPFT [1]) improves SatViT-V2's performance by another 1.2% on in-distribution and 2.4% on out-of-distribution data. Next, we find that pretrained RS transformers are better calibrated under distribution shifts than non-pretrained models and leveraging LPFT results in further improvements in model calibration. Lastly, we find that five measures of distribution shift are moderately correlated with biome transfer performance. We share code and pretrained model weights. (https://github.com/antofuller/SatViT)

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑