ViTGaze:基于 Vision Transformer 中交互特征的 Gaze Following
ViTGaze: Gaze Following with Interaction Features in Vision Transformers
School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);School of Future Technology, South China University of Technology(华南理工大学未来技术学院)
Gaze following aims to interpret human-scene interactions by predicting the person's focal point of gaze. Prevailing approaches often adopt a two-stage framework, whereby multi-modality information is extracted in the initial stage for gaze target prediction. Consequently, the efficacy of these methods highly depends on the precision of the preceding modality extraction. Others use a single-modality approach with complex decoders, increasing network computational load. Inspired by the remarkable success of pre-trained plain vision transformers (ViTs), we introduce a novel single-modality gaze following framework called ViTGaze. In contrast to previous methods, it creates a novel gaze following framework based mainly on powerful encoders (relative decoder parameters less than 1%). Our principal insight is that the inter-token interactions within self-attention can be transferred to interactions between humans and scenes. Leveraging this presumption, we formulate a framework consisting of a 4D interaction encoder and a 2D spatial guidance module to extract human-scene interaction information from self-attention maps. Furthermore, our investigation reveals that ViT with self-supervised pre-training has an enhanced ability to extract correlation information. Many experiments have been conducted to demonstrate the performance of the proposed method. Our method achieves state-of-the-art (SOTA) performance among all single-modality methods (3.4% improvement in the area under curve (AUC) score, 5.1% improvement in the average precision (AP)) and very comparable performance against multi-modality methods with 59% number of parameters less.
Xi’an University of Architecture and Technology(西安建筑科技大学);Beijing Institute of Technology(北京理工大学);University of Science and Technology of China(中国科学技术大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation
Department of Computer Science, University of Texas at Dallas, Richardson, TX, USA.(德克萨斯大学达拉斯分校计算机科学系);College of Computing, Georgia Institute of Technology, Atlanta, GA, USA(佐治亚理工学院计算机学院);Department of Computer Science, University of Wisconsin-Madison, Madison, WI, USA(威斯康星大学麦迪逊分校计算机科学系);Allen School of Computer Science, University of Washington, USA(华盛顿大学阿伦计算机科学学院)
Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer
Department of Computational Engineering, Lappeenranta-Lahti University of Technology LUT, Finland(拉彭兰塔-拉赫蒂理工大学计算工程系,芬兰);Faculty of Information Technology, Brno University of Technology, Czech Republic(布尔诺理工大学信息技术学院,捷克共和国)
GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
AIPM, School of Artificial Intelligence, Guilin University of Electronic Technology, China(AIPM,人工智能学院,桂林电子科技大学,中国);Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Canada(影像、视觉与人工智能实验室,École de Technologie Supérieure,加拿大)
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China(桂林电子科技大学人工智能学院);The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada(图像、视觉与人工智能实验室,蒙特利尔大学高级技术学院);Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia(计算机与信息科学学院信息科技系,沙特国王大学);Department of Cybersecurity, College of Computer Science and Engineering, Taibah University, Medina, Saudi Arabia(计算机科学与工程学院网络安全系,塔伊卜大学);Department of Networked Engineering, College of Computer Science and Engineering, Jeddah University, Jeddah, Saudi Arabia(计算机科学与工程学院网络工程系,吉达大学)
Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning
Northwestern Polytechnical University(西北工业大学);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Athens Academy(雅典学院);Shanghai Jiao Tong University(上海交通大学);ShanghaiTech University(上海科技大学)
Human-Object Interaction Prediction in Videos through Gaze Following
Technical University of Munich (TUM)(慕尼黑工业大学);Technische Universität Wien (TU Wien)(维也纳工业大学);Ulsan National Institute of Science and Technology (UNIST)(蔚山国立科学技术院);German Aerospace Center (DLR)(德国航空航天中心)
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia(佐治亚大学计算学院);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Georgia(佐治亚大学);Department of Physics and Astronomy(物理与天文学系)
TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Anhui University(安徽大学);United Arab Emirates University(阿联酋大学)
Vision-language models for decoding provider attention during neonatal resuscitation
Johns Hopkins University Applied Physics Laboratory(约翰斯·霍普金斯大学应用物理实验室);University of Pennsylvania Perelman School of Medicine(宾夕法尼亚大学佩雷尔曼医学院);Children’s Hospital of Philadelphia(费城儿童医院);University of Pennsylvania(宾夕法尼亚大学)
A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps
Institut de Neurosciences de la Timone, Aix-Marseille Université, CNRS(神经科学蒂姆研究所,艾克斯-马赛大学,CNRS);Laboratoire d’Électronique, Antennes et Télécommunications, Université Côte d’Azur, CNRS(电子、天线和电信实验室,蔚蓝海岸大学,CNRS)
HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination
University of Stuttgart Germany(斯图加特大学);University of Tuebingen Germany(图宾根大学);The Center for Bionic Intelligence Tuebingen Stuttgart Germany(图宾根-斯图加特生物智能中心)
DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection
School of Computation, Information and Technology, Technical University of Munich(计算信息与技术学院,慕尼黑技术大学);Department of Informatics, Technical University of Munich(信息学院,慕尼黑技术大学);University Research and Innovation Center, Obuda University(奥布达大学研究与创新中心);State Key Laboratory for Novel Software Technology, Nanjing University Suzhou Campus(新型软件技术国家重点实验室,南京大学苏州校区);School of Science and Technology, Nanjing University Suzhou Campus(科学与技术学院,南京大学苏州校区);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学)
Northwestern Polytechnical University(西北工业大学);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);Athens Academy(雅典学院);ShanghaiTech University(上海科技大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
Brown University(布朗大学);Columbia University(哥伦比亚大学);Emory University(埃默里大学);Johns Hopkins University(约翰霍普金斯大学);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);University of Michigan(密歇根大学);UC San Diego(圣地亚哥大学)
Pohang University of Science and Technology (POSTECH)(浦项科技大学);Microsoft Research Asia (MSRA)(微软亚洲研究院);University of Science and Technology of China (USTC)(中国科学技术大学)