arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2312.17117cs.CVcs.AI

Grounding-Prompter:利用多模态信息提示大语言模型实现长视频中的时间句子定位

Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videos

Tsinghua University(清华大学)

Houlun Chen, Xin Wang, Hong Chen, Zihan Song, Jia Jia, Wenwu Zhu

AI总结:

提出Grounding-Prompter方法,通过多模态信息提示大语言模型,并采用边界感知提示策略(含多尺度去噪思维链、有效性原则和一次性上下文学习),解决长视频中时间句子定位的复杂上下文与多模态挑战,实现最先进性能。

摘要:

时间句子定位(TSG)旨在根据给定的自然语言查询从视频中定位时刻,已引起广泛关注。现有工作主要针对短视频设计,无法处理长视频中的TSG,这带来了两个挑战:i) 长视频中的复杂上下文需要对更长的时刻序列进行时间推理,ii) 包括富含信息的文本语音在内的多种模态需要特殊设计以理解长视频内容。为应对这些挑战,本工作提出了一种Grounding-Prompter方法,通过利用多模态信息提示大语言模型(LLM)能够在长视频中执行TSG。具体而言,我们首先将TSG任务及其多模态输入(包括语音和视觉)转换为压缩的任务文本化形式。此外,为了增强复杂上下文下的时间推理,提出了一种边界感知提示策略,该策略包含三个方面:i) 我们设计了一种新颖的多尺度去噪思维链(CoT),逐步结合全局和局部语义并进行噪声过滤,ii) 我们设置了有效性原则,能够约束LLM按照特定格式生成合理的预测,iii) 我们引入了一次性上下文学习(ICL)通过模仿增强推理,提升LLM对TSG任务的理解。实验表明,我们的Grounding-Prompter方法达到了最先进的性能,揭示了利用多模态信息提示LLM对长视频中TSG的益处。

英文摘要:

Temporal Sentence Grounding (TSG), which aims to localize moments from videos based on the given natural language queries, has attracted widespread attention. Existing works are mainly designed for short videos, failing to handle TSG in long videos, which poses two challenges: i) complicated contexts in long videos require temporal reasoning over longer moment sequences, and ii) multiple modalities including textual speech with rich information require special designs for content understanding in long videos. To tackle these challenges, in this work we propose a Grounding-Prompter method, which is capable of conducting TSG in long videos through prompting LLM with multimodal information. In detail, we first transform the TSG task and its multimodal inputs including speech and visual, into compressed task textualization. Furthermore, to enhance temporal reasoning under complicated contexts, a Boundary-Perceptive Prompting strategy is proposed, which contains three folds: i) we design a novel Multiscale Denoising Chain-of-Thought (CoT) to combine global and local semantics with noise filtering step by step, ii) we set up validity principles capable of constraining LLM to generate reasonable predictions following specific formats, and iii) we introduce one-shot In-Context-Learning (ICL) to boost reasoning through imitation, enhancing LLM in TSG task understanding. Experiments demonstrate the state-of-the-art performance of our Grounding-Prompter method, revealing the benefits of prompting LLM with multimodal information for TSG in long videos.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑