arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.12686cs.CVcs.AI
收藏

用于动作检测的整体交互式Transformer网络

Holistic Interaction Transformer Network for Action Detection

National Tsing Hua University(国立清华大学);Microsoft AI R&D Center(微软人工智能研发中心)

Gueter Josmy Faure, Min-Hung Chen, Shang-Hong Lai

摘要:

行动是关于我们如何与环境互动,包括其他人,物体和我们自己。在本文中,我们提出了一种新的多模态整体交互Transformer网络(HIT),利用了在很大程度上被忽视,但关键的手和姿势的信息,大多数人的行动。建议的“HIT”网络是一个全面的双模态框架,包括RGB流和姿势流。它们中的每一个都分别对人、物体和手的交互进行建模。在每个子网络中,引入了一个内部模态聚合模块(IMA),该模块有选择地合并各个交互单元。然后使用注意力融合机制(AFM)将每个模态的结果特征粘合在一起。最后,我们从时间上下文中提取线索,以更好地分类使用缓存内存发生的动作。我们的方法在J-HMDB,UCF 101 -24和MultiSports数据集上的性能明显优于以前的方法。我们在AVA上也取得了有竞争力的成绩。该代码将在https://github.com/joslefaure/HIT上提供。

英文摘要:

Actions are about how we interact with the environment, including other people, objects, and ourselves. In this paper, we propose a novel multi-modal Holistic Interaction Transformer Network (HIT) that leverages the largely ignored, but critical hand and pose information essential to most human actions. The proposed "HIT" network is a comprehensive bi-modal framework that comprises an RGB stream and a pose stream. Each of them separately models person, object, and hand interactions. Within each sub-network, an Intra-Modality Aggregation module (IMA) is introduced that selectively merges individual interaction units. The resulting features from each modality are then glued using an Attentive Fusion Mechanism (AFM). Finally, we extract cues from the temporal context to better classify the occurring actions using cached memory. Our method significantly outperforms previous approaches on the J-HMDB, UCF101-24, and MultiSports datasets. We also achieve competitive results on AVA. The code will be available at https://github.com/joslefaure/HIT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑