利用 VLM 的可定位性与语义进行开放词汇动作检测
Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection
AI总结：
针对开放词汇动作检测问题，提出 OpenMixer 方法，利用大型视觉语言模型的语义和可定位性，结合 DETR 实现端到端检测，在已见和未见动作上均优于基线。
摘要：
动作检测旨在对视频中的人类动作进行空间和时间上的检测（识别与定位）。现有方法主要关注闭集设置，即动作检测器在固定动作类别的视频上进行训练和测试。然而，这种受限设置在开放世界中并不可行，因为测试视频不可避免地会超出已训练的动作类别。在本文中，我们解决了实用且具挑战性的开放词汇动作检测（OVAD）问题。其目标是在固定动作类别上训练模型的同时，检测测试视频中的任意动作。为实现这种开放词汇能力，我们提出了一种新方法 OpenMixer，在基于查询的检测 Transformer（DETR）家族中利用大型视觉语言模型（VLM）固有的语义和可定位性。具体而言，OpenMixer 由空间和时间 OpenMixer 模块（S-OMB 和 T-OMB）以及动态融合对齐（DFA）模块构建而成。这三个组件共同享有预训练 VLM 带来的强泛化能力以及 DETR 设计带来的端到端学习优势。此外，我们在多种设置下建立了 OVAD 基准，实验结果表明 OpenMixer 在检测已见和未见动作方面均优于基线方法。我们在 https://github.com/Cogito2012/OpenMixer 发布了代码、模型和数据集划分。
英文摘要：
Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes, models, and dataset splits at https://github.com/Cogito2012/OpenMixer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
