arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.08282cs.CV
收藏

LLaVA-ST:一种用于细粒度时空理解的多模态大语言模型

LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding

School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Meituan(美团)

Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, Si Liu

AI总结:

为解决多模态大模型难以同时进行细粒度时空定位的问题,本文提出 LLaVA-ST,通过语言对齐位置嵌入和时空打包器实现细粒度时空对齐,并构建 ST-Align 数据集与基准,在 11 个时空理解基准上取得优异性能。

摘要:

近年来,多模态大语言模型(MLLMs)取得了令人鼓舞的进展,但现有方法仍难以同时有效地处理时间定位和空间定位。这一挑战源于两个关键问题:第一,引入时空定位会产生大量坐标组合,使语言坐标表示与视觉坐标表示的对齐变得复杂;第二,在视频特征压缩过程中编码细粒度的时间和空间信息本身就很困难。为解决这些问题,我们提出了 LLaVA-ST,一种用于细粒度时空多模态理解的 MLLM。在 LLaVA-ST 中,我们提出了语言对齐位置嵌入(Language-Aligned Positional Embedding),将文本坐标特殊 token 嵌入到视觉空间中,从而简化细粒度时空对应关系的对齐。此外,我们设计了时空打包器(Spatial-Temporal Packer),将时间分辨率和空间分辨率的特征压缩解耦为两个不同的点到区域注意力处理流。进一步地,我们提出了包含 430 万个训练样本的 ST-Align 数据集,用于细粒度时空多模态理解。借助 ST-Align,我们提出了一种渐进式训练流程,通过从粗到细的连续阶段对齐视觉和文本特征。另外,我们引入了 ST-Align 基准来评估时空交错的细粒度理解任务,包括时空视频定位(STVG)、事件定位与描述(ELC)以及空间视频定位(SVG)。LLaVA-ST 在 11 个需要细粒度时间、空间或时空交错多模态理解的基准上取得了优异性能。我们的代码、数据和基准将在 https://github.com/appletea233/LLaVA-ST 发布。

英文摘要:

Recent advancements in multimodal large language models (MLLMs) have shown promising results, yet existing approaches struggle to effectively handle both temporal and spatial localization simultaneously. This challenge stems from two key issues: first, incorporating spatial-temporal localization introduces a vast number of coordinate combinations, complicating the alignment of linguistic and visual coordinate representations; second, encoding fine-grained temporal and spatial information during video feature compression is inherently difficult. To address these issues, we propose LLaVA-ST, a MLLM for fine-grained spatial-temporal multimodal understanding. In LLaVA-ST, we propose Language-Aligned Positional Embedding, which embeds the textual coordinate special token into the visual space, simplifying the alignment of fine-grained spatial-temporal correspondences. Additionally, we design the Spatial-Temporal Packer, which decouples the feature compression of temporal and spatial resolutions into two distinct point-to-region attention processing streams. Furthermore, we propose ST-Align dataset with 4.3M training samples for fine-grained spatial-temporal multimodal understanding. With ST-align, we present a progressive training pipeline that aligns the visual and textual feature through sequential coarse-to-fine stages.Additionally, we introduce an ST-Align benchmark to evaluate spatial-temporal interleaved fine-grained understanding tasks, which include Spatial-Temporal Video Grounding (STVG) , Event Localization and Captioning (ELC) and Spatial Video Grounding (SVG). LLaVA-ST achieves outstanding performance on 11 benchmarks requiring fine-grained temporal, spatial, or spatial-temporal interleaving multimodal understanding. Our code, data and benchmark will be released at Our code, data and benchmark will be released at https://github.com/appletea233/LLaVA-ST .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑