In this paper we introduce the Temporo-Spatial Vision Transformer (TSViT), a fully-attentional model for general Satellite Image Time Series (SITS) processing based on the Vision Transformer (ViT). TSViT splits a SITS record into non-overlapping patches in space and time which are tokenized and subsequently processed by a factorized temporo-spatial encoder. We argue, that in contrast to natural images, a temporal-then-spatial factorization is more intuitive for SITS processing and present experimental evidence for this claim. Additionally, we enhance the model's discriminative power by introducing two novel mechanisms for acquisition-time-specific temporal positional encodings and multiple learnable class tokens. The effect of all novel design choices is evaluated through an extensive ablation study. Our proposed architecture achieves state-of-the-art performance, surpassing previous approaches by a significant margin in three publicly available SITS semantic segmentation and classification datasets. All model, training and evaluation codes are made publicly available to facilitate further research.
Satellite Image Time Series Semantic Change Detection: Novel Architecture and Analysis of Domain Shift
LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France(法国国立桥梁学校 Gustave Eiffel大学 CNRS LIGM实验室);Inria Paris(法国国家信息与自动化研究所巴黎中心);Department of Computer Science, Ecole normale supérieure (ENS-PSL, CNRS, Inria)(巴黎文理研究大学 巴黎高等师范学院 计算机科学系);Courant Institute of Mathematical Sciences and Center for Data Science, New York University(纽约大学柯朗数学科学研究所与数据科学中心)
SITSMamba for Crop Classification based on Satellite Image Time Series
Wuhan University(武汉大学);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing(遥感信息工程国家重点实验室);School of Remote Sensing and Information Engineering(遥感信息工程学院)
On the use of Graphs for Satellite Image Time Series
Université Bretagne Sud(布列塔尼南部大学);IRISA(信息与自动化研究所);UMR CNRS 6074(法国国家科学研究中心6074号研究单位);Institut Universitaire de France (IUF)(法国大学研究院);Centre National d’Études Spatiales (CNES)(国家空间研究中心);UiT The Arctic University of Norway(北莫斯克大学)
Monitoring Pasture Restoration from Satellite Image Time Series: Caveats and Opportunities
Lund University(隆德大学);RISE Research Institutes of Sweden(瑞典RISE研究院);Climate AI Nordics(北欧气候人工智能中心);Swedish Centre for Impacts of Climate Extremes (CLIMES)(瑞典气候极端影响中心(CLIMES))
Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data
AMBER -- Advanced SegFormer for Multi-Band Image Segmentation: an application to Hyperspectral Imaging
University of Naples Federico II(那不勒斯费德里科二世大学);INFN – Section of Naples(国家核物理研究所那不勒斯分部);Stazione Zoologica Anton Dohrn(安托万·多尔恩动物站);INAF - Osservatorio Astronomico di Capodimonte(意大利国家天文研究所卡波迪蒙特天文台)
Pixel-wise Agricultural Image Time Series Classification: Comparisons and a Deformable Prototype-based Approach
LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS(LIGM实验室、巴黎路桥学校、古斯塔夫·埃菲尔大学、法国国家科学研究中心);Inria Paris(巴黎信息与自动化研究所);Ecole normale superieure (ENS-PSL, CNRS, Inria)(巴黎高等师范学院(PSL大学成员)、法国国家科学研究中心、法国信息与自动化研究所);Center for Data Science, New York University(纽约大学数据科学中心)
Pan-Arctic Permafrost Landform and Human-built Infrastructure Feature Detection with Vision Transformers and Location Embeddings
University of Connecticut(康涅狄格大学);Google(谷歌);Woodwell Climate Research Center(伍德韦尔气候研究中心);National Center for Supercomputing Applications, University of Illinois(伊利诺伊大学国家超级计算应用中心)
TAMMs: Change Understanding and Forecasting in Satellite Image Time Series with Temporal-Aware Multimodal Models
School of Computer Science & Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);School of Computing, National University of Singapore(计算学院,新加坡国立大学)
Detecting Looted Archaeological Sites from Satellite Image Time Series
Iconem(Iconem公司);DAFA, French archaeological delegation in Afghanistan(法国驻阿富汗考古代表团);Department of Computer Science, Ecole normale supérieure (ENS-PSL, CNRS, Inria)(巴黎高等师范学院计算机科学系(ENS-PSL、法国国家科学研究中心、法国国家信息与自动化研究所));Courant Institute of Mathematical Sciences and Center for Data Science, New York University(纽约大学柯朗数学科学研究所与数据科学中心)
Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuning
Department of Information Technology Uppsala University(信息科技系乌普萨拉大学);Department of Immunology, Genetics and Pathology Uppsala University(免疫学、遗传学和病理学系乌普萨拉大学)
Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Research and Early Development (gRED), Genentech(基因泰克研发与早期开发部);Biology Research — AI Development (BRAID), Genentech(基因泰克生物学研究——人工智能开发部)
Spatio-Temporal Transformers for Long-Term NDVI Forecasting
Dept. of Computer Science, Bar-Ilan University(巴伊兰大学计算机科学系);Dept. of Computer Science, College of Law(法律学院计算机科学系);Faculty of Civil and Environmental Engineering, Technion Israel Institute of Technology(技术学院环境与土木工程学院)
SceneMixer: Exploring Convolutional Mixing Networks for Remote Sensing Scene Classification
College of Engineering(工程学院);University of Dubai(迪拜大学);Department of Geography(地理系);Simon Fraser University(西蒙弗雷泽大学);Department of Computer Science(计算机科学系);Tezpur University(泰朱普大学)