Gaze understanding unifies the detection of people, their gaze targets, and objects of interest into a single framework, offering critical insight into visual attention and intent estimation. Although prior research has modelled gaze cues in visual scenes, a unified system is still needed for gaze understanding using both visual and language prompts. This paper introduces GazeVLM, a novel Vision-Language Model (VLM) for multi-task gaze understanding in images, addressing person detection, gaze target detection, and gaze object identification. While other transformer-based methods exist for gaze analysis, GazeVLM represents, to our knowledge, the first application of a VLM to these combined tasks, allowing for selective execution of each task. Through the integration of visual (RGB and depth) and textual modalities, our ablation study on visual input combinations revealed that a fusion of RGB images with HHA-encoded depth maps, guided by text prompts, yields superior performance. We also introduce an object-level gaze detection metric for gaze object identification ($AP_{ob}$). Through experiments, GazeVLM demonstrates significant improvements, notably achieving state-of-the-art evaluation scores on GazeFollow and VideoAttentionTarget datasets.
Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Computer Science and Engineering, University of Texas, Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Massachusetts General Hospital, Harvard Medical School（麻省总医院哈佛医学院）；Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）
Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Bioengineering Department and Imperial-X（生物工程系和Imperial-X）；Imperial College London（帝国理工学院伦敦分校）；College of physics and information engineering, Fuzhou University（福州大学物理与信息工程学院）；Department of Surgery and Cancer, Imperial College London（外科与癌症部门，帝国理工学院伦敦分校）；National Heart and Lung Institute, Imperial College London（国家心脏和肺研究所，帝国理工学院伦敦分校）；Cardiovascular Research Centre, Royal Brompton Hospital（心血管研究中心，皇家布里顿医院）；School of Biomedical Engineering & Imaging Sciences, King’s College London（生物医学工程与成像科学学院，国王学院伦敦分校）
Vision-language models for decoding provider attention during neonatal resuscitation
Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；University of Pennsylvania Perelman School of Medicine（宾夕法尼亚大学佩雷尔曼医学院）；Children’s Hospital of Philadelphia（费城儿童医院）；University of Pennsylvania（宾夕法尼亚大学）
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
Brown University（布朗大学）；Columbia University（哥伦比亚大学）；Emory University（埃默里大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；UC San Diego（圣地亚哥大学）
From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation
Department of Engineering Science, University of Oxford（工程科学系，牛津大学）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Information Science and Technology, Northwest University（信息科学与技术学院，西北大学）
ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation
Department of Computer Science, University of Texas at Dallas, Richardson, TX, USA.（德克萨斯大学达拉斯分校计算机科学系）；College of Computing, Georgia Institute of Technology, Atlanta, GA, USA（佐治亚理工学院计算机学院）；Department of Computer Science, University of Wisconsin-Madison, Madison, WI, USA（威斯康星大学麦迪逊分校计算机科学系）；Allen School of Computer Science, University of Washington, USA（华盛顿大学阿伦计算机科学学院）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art obje…
From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）；Surrey Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）
From Scene to Object: Text-Guided Dual-Gaze Prediction
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology（Cho Chun Shik 移动研究生院，韩国科学技术院）
TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Anhui University（安徽大学）；United Arab Emirates University（阿联酋大学）
Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm
Key Laboratory of Pervasive Computing, Tsinghua University（清华大学普适计算重点实验室）；The University of New South Wales（新南威尔士大学）；SKLSDE Lab, Beihang University（北航SKLSDE实验室）
Joint attention is a core, early-developing form of social interaction. It is based on our ability to discriminate the third party objects that other people are looking at. While it has been shown that people can accurately determine whether another person is looking directly at them versus away, little is known about human ability to discriminate a third person gaze directed towards objects that are further away, especially in unconstraint cases where the looker can move her head and eyes freely. In this paper we address this question by jointly exploring human psychophysics and a cognitively…
Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following
Beijing Jiaotong University（北京交通大学）；University of Birmingham（英国伯明翰大学）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；Microsoft, UK（微软公司（英国））
Augmented Reality (AR) see-through vision is an interesting research topic since it enables users to see through a wall and see the occluded objects. Most existing research focuses on the visual effects of see-through vision, while the interaction method is less studied. However, we argue that using common interaction modalities, e.g., midair click and speech, may not be the optimal way to control see-through vision. This is because when we want to see through something, it is physically related to our gaze depth/vergence and thus should be naturally controlled by the eyes. Following this idea…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiaotong University（上海交通大学）
Recently, there emerges a series of vision Transformers, which show superior performance with a more compact model size than conventional convolutional neural networks, thanks to the strong ability of Transformers to model long-range dependencies. However, the advantages of vision Transformers also come with a price: Self-attention, the core part of Transformer, has a quadratic complexity to the input sequence length. This leads to a dramatic increase of computation and memory cost with the increase of sequence length, thus introducing difficulties when applying Transformers to the vision task…
Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model
Xi’an University of Architecture and Technology（西安建筑科技大学）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）
By and large, existing computational models of visual attention tacitly assume perfect vision and full access to the stimulus and thereby deviate from foveated biological vision. Moreover, modeling top-down attention is generally reduced to the integration of semantic features without incorporating the signal of a high-level visual tasks that have been shown to partially guide human attention. We propose the Neural Visual Attention (NeVA) algorithm to generate visual scanpaths in a top-down manner. With our method, we explore the ability of neural networks on which we impose a biologically-ins…
Recent work has proven the effectiveness of transformers in many computer vision tasks. However, the performance of transformers in gaze estimation is still unexplored. In this paper, we employ transformers and assess their effectiveness for gaze estimation. We consider two forms of vision transformer which are pure transformers and hybrid transformers. We first follow the popular ViT and employ a pure transformer to estimate gaze from images. On the other hand, we preserve the convolutional layers and integrate CNNs as well as transformers. The transformer serves as a component to complement …
