基于内容条件查询的Transformer的视频指代表表达式理解
Video Referring Expression Comprehension via Transformer with Content-conditioned Query
AI总结：
针对视频指代表达式理解中可学习查询难以表征开放语义的问题，提出ConFormer模型，通过基于视频和语言内容的动态查询及跨模态对齐设计，显著提升了目标定位性能。
摘要：
视频指代表表达式理解（Video REC）旨在根据查询的自然语言在视频中定位目标对象。近期，基于Transformer的方法与可学习查询的改进提升了了视频REC的性能。然而，我们认为这种朴素的查询设计并不理想，因为视频REC由文本监督带来的开放放特性，使得仅靠少数缓慢更新的新的查询不足以表征众多潜在的语义类别。我们提出创建动态查询的解决方案是使其其动态查询根据输入视频和语言进行条件建模。模型所引用的各种对象。我们在放置固定数量的可学习边界框（bounding boxes，作为输入视频和语言特征提供先验信息。此外，当前查询特征忽略了跨模态对齐的重要性。为解决此问题，我们提出名为ConFormer的的模型，在广泛基准数据集上达到了8.75%绝对提升。在Acc.@0.6比较的之前人状态至艺术模型。
英文摘要：
Video Referring Expression Comprehension (REC) aims to localize a target object in videos based on the queried natural language. Recent improvements in video REC have been made using Transformer-based methods with learnable queries. However, we contend that this naive query design is not ideal given the open-world nature of video REC brought by text supervision. With numerous potential semantic categories, relying on only a few slow-updated queries is insufficient to characterize them. Our solution to this problem is to create dynamic queries that are conditioned on both the input video and language to model the diverse objects referred to. Specifically, we place a fixed number of learnable bounding boxes throughout the frame and use corresponding region features to provide prior information. Also, we noticed that current query features overlook the importance of cross-modal alignment. To address this, we align specific phrases in the sentence with semantically relevant visual areas, annotating them in existing video datasets (VID-Sentence and VidSTG). By incorporating these two designs, our proposed model (called ConFormer) outperforms other models on widely benchmarked datasets. For example, in the testing split of VID-Sentence dataset, ConFormer achieves 8.75% absolute improvement on Accu.@0.6 compared to the previous state-of-the-art model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
