Foundation models have revolutionized artificial intelligence, setting new benchmarks in performance and enabling transformative capabilities across a wide range of vision and language tasks. However, despite the prevalence of spatio-temporal data in critical domains such as transportation, public health, and environmental monitoring, spatio-temporal foundation models (STFMs) have not yet achieved comparable success. In this paper, we articulate a vision for the future of STFMs, outlining their essential characteristics and the generalization capabilities necessary for broad applicability. We critically assess the current state of research, identifying gaps relative to these ideal traits, and highlight key challenges that impede their progress. Finally, we explore potential opportunities and directions to advance research towards the aim of effective and broadly applicable STFMs.
A Decade of Deep Learning for Remote Sensing Spatiotemporal Fusion: Advances, Challenges, and Opportunities
School of Computer Science, China University of Geosciences (Wuhan)（中国地质大学（武汉）计算机科学学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology (HITSZ)（哈尔滨工业大学（深圳））；School of Computer Science, Peking University（北京大学计算机学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
4D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimation
National Key Lab of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（多谱信息智能处理国家实验室，人工智能与自动化学院，华中科技大学）；School of Computing, National University of Singapore（计算学院，新加坡国立大学）
Non-Rigid 3D Shape Correspondences: From Foundations to Open Challenges and Opportunities
MPI for Informatics（马克斯·普朗克信息学研究所）；University of Bonn（波恩大学）；Lamarr Institute（拉马尔研究所）；Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学理工学研究所）；Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Foundation Models in Computational Pathology: A Review of Challenges, Opportunities, and Impact
Prince Sattam bin Abdulaziz University（萨塔姆·本·阿卜杜勒阿齐兹王子大学）；Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）；University of Warwick（华威大学）；Frontier AI（前沿人工智能）
AgriFM: A Multi-source Temporal Remote Sensing Foundation Model for Agriculture Mapping
Jockey Club STEM Lab of Quantitative Remote Sensing, Department of Geography, The University of Hong Kong, Hong Kong, China（香港大学地理系金钟STEM实验室）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University, Beijing, China（北航航天智能科学与技术学院）；School of Remote Sensing and Information Engineering, Wuhan University, China（武汉大学遥感与信息工程学院）
UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines
MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiaotong University（上海交通大学）；Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong（香港中文大学信兴高等工程研究所）
University of Minnesota（明尼苏达大学）；University of Pittsburgh（匹兹堡大学）
The availability of massive earth observing satellite data provide huge opportunities for land use and land cover mapping. However, such mapping effort is challenging due to the existence of various land cover classes, noisy data, and the lack of proper labels. Also, each land cover class typically has its own unique temporal pattern and can be identified only during certain periods. In this article, we introduce a novel architecture that incorporates the UNet structure with Bidirectional LSTM and Attention mechanism to jointly exploit the spatial and temporal nature of satellite data and to b…
The task of Video Question Answering (VideoQA) consists in answering natural language questions about a video and serves as a proxy to evaluate the performance of a model in scene sequence understanding. Most methods designed for VideoQA up-to-date are end-to-end deep learning architectures which struggle at complex temporal and causal reasoning and provide limited transparency in reasoning steps. We present the HySTER: a Hybrid Spatio-Temporal Event Reasoner to reason over physical events in videos. Our model leverages the strength of deep learning methods to extract information from video fr…
Large Language Models and Foundation Models in Smart Agriculture: Basics, Opportunities, and Challenges
Michigan State University（密歇根州立大学）；Jeonbuk National University（国立全北大学）；North Carolina State University（北卡罗来纳州立大学）；Southeast University（东南大学）；Nanyang Technological University（南洋理工大学）
