变分跨图推理与自适应结构化语义学习用于组合时间定位
Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
AI总结：
针对现有时间定位模型组合泛化能力不足的问题，本文提出新任务与数据集划分，并采用变分跨图推理及自适应结构化语义学习，实现视频与语言层次化语义图的细粒度对应，显著提升组合泛化性能。
摘要：
时间定位是根据查询语句在未修剪的视频中定位特定片段的任务。该任务在计算机视觉领域已获得显著发展势头，因为它利用自然语言描述的语义多样性，能够超越预定义的活动类别进行活动定位。语义多样性源于语言学中的组合性原则，即通过以新颖方式组合已知词汇，可以系统地描述新语义（组合泛化）。然而，现有的时间定位数据集并未精心设计以评估组合泛化能力。为系统性地基准测试时间定位模型的组合泛化能力，我们引入了一个新的组合时间定位任务，并构建了两个新的数据集划分，即Charades-CG和ActivityNet-CG。在我们的新数据集划分上评估最先进方法时，我们实证发现它们无法泛化到包含已知词汇新颖组合的查询。我们认为视频和语言内部固有的结构化语义是实现组合泛化的关键因素。基于这一见解，我们提出了一种变分跨图推理框架，该框架分别将视频和语言显式分解为层次化语义图，并学习两个图之间的细粒度语义对应关系。此外，我们引入了一种新颖的自适应结构化语义学习方法，以推导出结构感知且领域可泛化的图表示，从而促进两个图之间的细粒度语义对应推理。大量实验验证了我们方法优越的组合泛化能力。
英文摘要：
Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. This task has achieved significant momentum in the computer vision community as it enables activity grounding beyond pre-defined activity classes by utilizing the semantic diversity of natural language descriptions. The semantic diversity is rooted in the principle of compositionality in linguistics, where novel semantics can be systematically described by combining known words in novel ways (compositional generalization). However, existing temporal grounding datasets are not carefully designed to evaluate the compositional generalizability. To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i.e., Charades-CG and ActivityNet-CG. When evaluating the state-of-the-art methods on our new dataset splits, we empirically find that they fail to generalize to queries with novel combinations of seen words. We argue that the inherent structured semantics inside the videos and language is the crucial factor to achieve compositional generalization. Based on this insight, we propose a variational cross-graph reasoning framework that explicitly decomposes video and language into hierarchical semantic graphs, respectively, and learns fine-grained semantic correspondence between the two graphs. Furthermore, we introduce a novel adaptive structured semantics learning approach to derive the structure-informed and domain-generalizable graph representations, which facilitate the fine-grained semantic correspondence reasoning between the two graphs. Extensive experiments validate the superior compositional generalizability of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
