面向语言引导视频分割的局部-全局上下文感知 Transformer
Local-Global Context Aware Transformer for Language-Guided Video Segmentation
AI总结：
本文提出局部-全局上下文感知 Transformer Locater，以有限记忆实现语言表达式对整段视频的线性高效查询，并构建 A2D-S+ 数据集，实验显示其达到最先进性能。
摘要：
我们探究了语言引导视频分割（LVS）任务。以往算法大多采用 3D CNN 学习视频表征，难以捕捉长期上下文，并且容易受到视觉-语言错位的影响。鉴于此，我们提出 Locater（local-global context aware Transformer，局部-全局上下文感知 Transformer），它通过有限记忆增强 Transformer 架构，从而以高效方式利用语言表达式查询整个视频。该记忆被设计为包含两个组成部分——一个用于持久保存全局视频内容，另一个用于动态收集局部时间上下文和分割历史。基于记忆中的局部-全局上下文以及每一帧的特定内容，Locater 全面而灵活地理解表达式，将其转化为适用于每一帧的自适应查询向量。该向量用于查询相应帧以生成掩码。该记忆还使 Locater 能够以线性时间复杂度和恒定大小的记忆处理视频，而 Transformer 式自注意力计算会随序列长度呈二次方增长。为全面检验 LVS 模型的视觉定位能力，我们贡献了一个新的 LVS 数据集 A2D-S+，它构建于 A2D-S 数据集之上，但在区分相似对象方面提出了更高挑战。在三个 LVS 数据集以及我们的 A2D-S+ 上的实验表明，Locater 优于以往的最先进方法。此外，我们在第三届大规模视频对象分割挑战赛的指代表视频对象分割赛道中获得第一名，Locater 是该获奖方案的基础。我们的代码和数据集可在 https://github.com/leonnnop/Locater 获取。
英文摘要：
We explore the task of language-guided video segmentation (LVS). Previous algorithms mostly adopt 3D CNNs to learn video representation, struggling to capture long-term context and easily suffering from visual-linguistic misalignment. In light of this, we present Locater (local-global context aware Transformer), which augments the Transformer architecture with a finite memory so as to query the entire video with the language expression in an efficient manner. The memory is designed to involve two components -- one for persistently preserving global video content, and one for dynamically gathering local temporal context and segmentation history. Based on the memorized local-global context and the particular content of each frame, Locater holistically and flexibly comprehends the expression as an adaptive query vector for each frame. The vector is used to query the corresponding frame for mask generation. The memory also allows Locater to process videos with linear time complexity and constant size memory, while Transformer-style self-attention computation scales quadratically with sequence length. To thoroughly examine the visual grounding capability of LVS models, we contribute a new LVS dataset, A2D-S+, which is built upon A2D-S dataset but poses increased challenges in disambiguating among similar objects. Experiments on three LVS datasets and our A2D-S+ show that Locater outperforms previous state-of-the-arts. Further, we won the 1st place in the Referring Video Object Segmentation Track of the 3rd Large-scale Video Object Segmentation Challenge, where Locater served as the foundation for the winning solution. Our code and dataset are available at: https://github.com/leonnnop/Locater
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
