arXiv 2510.164442026-10-06cs.CVcs.MMcs.ROeess.IV

RefAtomNet++：利用基于语义检索的多轨迹 Mamba 推进指称原子视频动作识别

RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Multi-Trajectory Semantic Retrieval

Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（人机化研究所，卡尔斯鲁厄技术大学）；RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）；School of Artificial Intelligence and Robotics（人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology（机器人视觉感知与控制技术国家工程研究中心）；Chinese University of Hong Kong（香港中文大学）；Shanghai AI Lab（上海人工智能实验室）

Kunyu Peng, Di Wen, Jia Fu, Jiamin Wu, Kailun Yang, Junwei Zheng, Ruiping Liu, Yufan Chen, Yuqian Fu, Danda Pani Paudel, Luc Van Gool, Rainer Stiefelhagen

备注：: Extended version of ECCV 2024 paper arXiv:2407.01872. The dataset and code are available at https://github.com/KPeng9510/refAVA2

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文扩展RefAVA++数据集并提出RefAtomNet++，通过语义对齐交叉注意力与多轨迹Mamba增强跨模态检索，提升语言引导的原子动作识别性能。

摘要：

指称原子视频动作识别（RAVAR）旨在根据自然语言描述，识别特定感兴趣人物的细粒度、原子级动作。与传统动作识别和检测任务不同，RAVAR强调精确的语言引导动作理解，这对于复杂多人场景中的交互式人类动作分析尤为关键。在本文中，我们将先前提出的RefAVA数据集扩展为RefAVA++，该数据集总共包含超过290万帧和超过7.51万个带标注人物。我们使用来自多个相关领域的基线方法对该数据集进行基准测试，包括原子动作定位、视频问答和文本-视频检索，以及我们早期的模型RefAtomNet。尽管RefAtomNet通过引入智能体注意力来突出显著特征而优于其他基线，但其对齐和检索跨模态信息的能力仍然有限，导致在定位目标人物和预测细粒度动作方面性能欠佳。为克服上述局限，我们提出RefAtomNet++，这是一种通过多层级语义对齐交叉注意力机制，并结合部分关键词、场景属性和完整句子层级的多轨迹Mamba建模来推进跨模态令牌聚合的新颖框架。具体而言，在部分关键词和场景属性层级，扫描轨迹通过在每个时间步动态选择最近的视觉空间令牌来构建。此外，我们设计了一种多层级语义对齐交叉注意力策略，能够更有效地聚合不同语义层级上的空间和时间令牌。实验表明，RefAtomNet++取得了新的最先进结果。数据集和代码发布于https://github.com/KPeng9510/refAVA2。

英文摘要：

Who is being described, where are they, and what are they doing? Referring Atomic Video Action Recognition (RAVAR) answers these questions jointly by grounding a natural-language reference to a person and recognizing that person's fine-grained atomic actions in complex multi-person videos. Progress in RAVAR is constrained by limited benchmark scale and weak alignment between fine-grained linguistic and scene cues and temporally coherent visual cues. We address both challenges with a new dataset and model. We introduce RefAVA++, a large-scale dataset comprising 2,950,920$ frames, 75,111 annotated person instances, and 80 atomic action categories. Its references describe appearance and spatial attributes while deliberately omitting action labels, requiring models to infer actions directly from visual cues. We further propose RefAtomNet++, which models complementary semantics at the holistic-sentence, partial-keyword, and scene-attribute levels. Fine-grained semantic cues retrieve aligned visual tokens across time to construct trajectories, which are aggregated through Mamba-based state-space modeling and fused using multi-hierarchical semantic-aligned cross-attention. This design enables accurate joint person localization and multi-label atomic action recognition. Equipped with the InternVideo2.5 backbone, RefAtomNet++ achieves 50.39%/51.14% mIoU, 59.21%/59.55% mAP, and 73.89%/76.59% AUROC on RefAVA, and 49.49%/50.33% mIoU, 62.33%/61.90% mAP, and 76.06%/76.10% AUROC on RefAVA++ validation/test sets, respectively. The dataset and code are available at https://github.com/KPeng9510/refAVA2.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。