利用多模态数据是理解地理对象的内在要求。然而,由于各类时空模态在结构和语义上具有高度异质性,多模态时空数据的联合解译长期以来一直是一个极具挑战性的问题。主要挑战在于在不同模态的内聚性与自主性之间取得平衡。随着模态数量增加,这种权衡会逐渐呈现非线性。受人类认知系统和语言哲学的启发,即来自五种感官的知觉信号汇聚到语言中,我们提出 Language as Reference Framework(LaRF,语言作为参考框架),这是构建多模态统一模型的一项基本原则。在此基础上,我们提出 AllSpark,一个多模态时空通用人工智能模型。我们的模型将十种不同模态整合到统一框架中。为实现模态内聚,AllSpark 引入模态桥和多模态大型语言模型(LLM),将多样的模态特征映射到语言特征空间。为保持模态自主性,AllSpark 使用模态专用编码器提取各类时空模态的 token。最后,观察到模型可解释性与下游任务之间存在差距,我们设计了模态专用提示和任务头,增强模型在特定任务上的泛化能力。实验表明,语言的引入使 AllSpark 无需额外训练即可在 RGB 和点云模态的少样本分类任务中表现优异,最高超过基线性能 41.82%。源代码可在 https://github.com/GeoX-Lab/AllSpark 获取。
英文摘要:
Leveraging multimodal data is an inherent requirement for comprehending geographic objects. However, due to the high heterogeneity in structure and semantics among various spatio-temporal modalities, the joint interpretation of multimodal spatio-temporal data has long been an extremely challenging problem. The primary challenge resides in striking a trade-off between the cohesion and autonomy of diverse modalities. This trade-off becomes progressively nonlinear as the number of modalities expands. Inspired by the human cognitive system and linguistic philosophy, where perceptual signals from the five senses converge into language, we introduce the Language as Reference Framework (LaRF), a fundamental principle for constructing a multimodal unified model. Building upon this, we propose AllSpark, a multimodal spatio-temporal general artificial intelligence model. Our model integrates ten different modalities into a unified framework. To achieve modal cohesion, AllSpark introduces a modal bridge and multimodal large language model (LLM) to map diverse modal features into the language feature space. To maintain modality autonomy, AllSpark uses modality-specific encoders to extract the tokens of various spatio-temporal modalities. Finally, observing a gap between the model's interpretability and downstream tasks, we designed modality-specific prompts and task heads, enhancing the model's generalization capability across specific tasks. Experiments indicate that the incorporation of language enables AllSpark to excel in few-shot classification tasks for RGB and point cloud modalities without additional training, surpassing baseline performance by up to 41.82\%. The source code is available at https://github.com/GeoX-Lab/AllSpark.
DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences(中国科学院大学杭州高等研究院);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Department of Computer Science and Engineering, University of North Texas(北卡罗来纳州立大学计算机科学与工程系);OPPO AI Center(OPPO人工智能中心)
DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems
Southeast University(东南大学);Jiangsu Province Collaborative Innovation Center of Modern Urban Traffic Technologies(江苏省现代城市交通技术协同创新中心);City University of Hong Kong(香港城市大学);School of Mathematics, Southeast University(东南大学数学学院);Systems Research Institute of the Polish Academy of Sciences(波兰科学院系统研究所);Luoyang Normal University(洛阳师范学院);Purple Mountain Laboratories(紫金山实验室);AGH University of Krakow(AGH科技大学)
Air Quality Prediction with A Meteorology-Guided Modality-Decoupled Spatio-Temporal Network
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Huawei Cloud Enterprise Intelligence(华为云企业智能)
Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer -- Current Trends and Research Perspectives
NASK - National Research Institute(国家研究 institute);Faculty of Mathematics and Information Science, Warsaw University of Technology(数学与信息科学学院,华沙技术大学);Faculty of Physics and Applied Computer Science, AGH University of Science and Technology(物理与应用计算机科学学院,AGH科技大学)
Three-Stream 3D/1D CNN for Fine-Grained Action Classification and Segmentation in Table Tennis
Max Planck Institute for Evolutionary Anthropology(马克斯·普朗克进化人类学研究所);Univ. Bordeaux(波尔多大学);CNRS(法国国家科学研究中心);Bordeaux INP(波尔多国立理工学院);LaBRI(波尔多计算机科学研究所);MIA, La Rochelle University(拉罗谢尔大学MIA研究所);IMS, University of Bordeaux(波尔多大学IMS研究所)
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
School of Automotive Studies, Tongji University(同济大学汽车学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Automobile, Chang'an University(长安大学汽车学院);School of Information and Electrical Engineering, Hangzhou City University(杭州城市学院信息与电气工程学院);College of Science and Engineering, James Cook University(詹姆斯库克大学科学与工程学院)
Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Zhongshan Hospital(中山医院);State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University(上海科技大学先进医用材料与器件国家重点实验室);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)
Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement
State Key Laboratory of Software Development Environment, Beihang University(北京航空航天大学软件开发环境国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);National Computer Network Emergency Response Technical Team, Coordination Center of China(中国国家计算机网络应急技术处理协调中心)