arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12778cs.CV
收藏

ViTGaze:基于 Vision Transformer 中交互特征的 Gaze Following

ViTGaze: Gaze Following with Interaction Features in Vision Transformers

School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);School of Future Technology, South China University of Technology(华南理工大学未来技术学院)

Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, Xiangmin Xu

AI总结:

提出单模态 gaze following 框架 ViTGaze,利用 ViT self-attention 中的 inter-token 交互提取人-场景交互信息,在减少参数的同时实现了 SOTA 性能。

摘要:

Gaze following 旨在通过预测人的注视焦点来解释人与场景的交互。现有方法通常采用两阶段框架,在初始阶段提取多模态信息以进行注视目标预测,因此其效果高度依赖于前序模态提取的精度。另一些方法使用单模态方法和复杂的解码器,增加了网络计算负担。受预训练 plain vision transformer (ViT) 成功的启发,我们提出了一种名为 ViTGaze 的单模态 gaze following 框架。与以往方法不同,它主要基于强大的编码器构建了新颖的框架(相对解码器参数不到 1%)。我们的主要洞察是,self-attention 中的 inter-token 交互可以转化为人与场景之间的交互。基于这一假设,我们构建了由 4D 交互编码器和 2D 空间指导模块组成的框架,从 self-attention map 中提取人-场景交互信息。此外,我们的研究揭示,具有 self-supervised pre-training 的 ViT 在提取相关信息方面能力更强。大量实验证明了所提方法的性能。该方法在所有单模态方法中实现了 state-of-the-art (SOTA) 性能(AUC 分数提升 3.4%,average precision (AP) 提升 5.1%),并且在参数量减少 59% 的情况下,与多模态方法相比具有极具竞争力的性能。

英文摘要:

Gaze following aims to interpret human-scene interactions by predicting the person's focal point of gaze. Prevailing approaches often adopt a two-stage framework, whereby multi-modality information is extracted in the initial stage for gaze target prediction. Consequently, the efficacy of these methods highly depends on the precision of the preceding modality extraction. Others use a single-modality approach with complex decoders, increasing network computational load. Inspired by the remarkable success of pre-trained plain vision transformers (ViTs), we introduce a novel single-modality gaze following framework called ViTGaze. In contrast to previous methods, it creates a novel gaze following framework based mainly on powerful encoders (relative decoder parameters less than 1%). Our principal insight is that the inter-token interactions within self-attention can be transferred to interactions between humans and scenes. Leveraging this presumption, we formulate a framework consisting of a 4D interaction encoder and a 2D spatial guidance module to extract human-scene interaction information from self-attention maps. Furthermore, our investigation reveals that ViT with self-supervised pre-training has an enhanced ability to extract correlation information. Many experiments have been conducted to demonstrate the performance of the proposed method. Our method achieves state-of-the-art (SOTA) performance among all single-modality methods (3.4% improvement in the area under curve (AUC) score, 5.1% improvement in the average precision (AP)) and very comparable performance against multi-modality methods with 59% number of parameters less.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑