arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00546cs.AIcs.LG
收藏

AllSpark:以语言为参考框架的十模态多模态时空通用智能模型

AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Framework

Central South University(中南大学);Xiangjiang Laboratory(湘江实验室);Southwest Jiaotong University(西南交通大学);Wuhan University(武汉大学);Peking University(北京大学);Huawei Technologies Co., Ltd(华为技术有限公司);BDS Micro Chip Inc(北斗微芯公司)

Run Shao, Cheng Yang, Qiujun Li, Qing Zhu, Yongjun Zhang, YanSheng Li, Yu Liu, Yong Tang, Dapeng Liu, Shizhong Yang, Haifeng Li

AI总结:

针对多模态时空数据联合解译难题,本文提出 LaRF 与 AllSpark 模型,以语言空间桥接并统一十种模态,在 RGB 和点云少样本分类中最高提升 41.82%。

摘要:

利用多模态数据是理解地理对象的内在要求。然而,由于各类时空模态在结构和语义上具有高度异质性,多模态时空数据的联合解译长期以来一直是一个极具挑战性的问题。主要挑战在于在不同模态的内聚性与自主性之间取得平衡。随着模态数量增加,这种权衡会逐渐呈现非线性。受人类认知系统和语言哲学的启发,即来自五种感官的知觉信号汇聚到语言中,我们提出 Language as Reference Framework(LaRF,语言作为参考框架),这是构建多模态统一模型的一项基本原则。在此基础上,我们提出 AllSpark,一个多模态时空通用人工智能模型。我们的模型将十种不同模态整合到统一框架中。为实现模态内聚,AllSpark 引入模态桥和多模态大型语言模型(LLM),将多样的模态特征映射到语言特征空间。为保持模态自主性,AllSpark 使用模态专用编码器提取各类时空模态的 token。最后,观察到模型可解释性与下游任务之间存在差距,我们设计了模态专用提示和任务头,增强模型在特定任务上的泛化能力。实验表明,语言的引入使 AllSpark 无需额外训练即可在 RGB 和点云模态的少样本分类任务中表现优异,最高超过基线性能 41.82%。源代码可在 https://github.com/GeoX-Lab/AllSpark 获取。

英文摘要:

Leveraging multimodal data is an inherent requirement for comprehending geographic objects. However, due to the high heterogeneity in structure and semantics among various spatio-temporal modalities, the joint interpretation of multimodal spatio-temporal data has long been an extremely challenging problem. The primary challenge resides in striking a trade-off between the cohesion and autonomy of diverse modalities. This trade-off becomes progressively nonlinear as the number of modalities expands. Inspired by the human cognitive system and linguistic philosophy, where perceptual signals from the five senses converge into language, we introduce the Language as Reference Framework (LaRF), a fundamental principle for constructing a multimodal unified model. Building upon this, we propose AllSpark, a multimodal spatio-temporal general artificial intelligence model. Our model integrates ten different modalities into a unified framework. To achieve modal cohesion, AllSpark introduces a modal bridge and multimodal large language model (LLM) to map diverse modal features into the language feature space. To maintain modality autonomy, AllSpark uses modality-specific encoders to extract the tokens of various spatio-temporal modalities. Finally, observing a gap between the model's interpretability and downstream tasks, we designed modality-specific prompts and task heads, enhancing the model's generalization capability across specific tasks. Experiments indicate that the incorporation of language enables AllSpark to excel in few-shot classification tasks for RGB and point cloud modalities without additional training, surpassing baseline performance by up to 41.82\%. The source code is available at https://github.com/GeoX-Lab/AllSpark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑