Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions. This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egocentric behavior understanding. Unlike existing methods that rely solely on visual data and overlook gaze information, our approach directly incorporates gaze information into the VLM architecture during training. By generating gaze-based queries, the model dynamically focuses on gaze-highlighted regions, while a gaze-regularization mechanism ensures the alignment of model attention with human attention patterns. To better understand how gaze can be effectively integrated into VLMs, we conducted extensive experiments exploring various strategies for incorporating gaze data. These innovations enable the prediction of future events with detailed action descriptions. Experimental results demonstrate a nearly 13 % improvement in semantic scores compared to baseline models not leveraging gaze data, highlighting the effectiveness of our approach. This work establishes a foundation for leveraging the human gaze in VLMs, significantly boosting their predictive capabilities in applications requiring accurate and robust future event prediction.
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation
Department of Computer Science, University of Texas at Dallas, Richardson, TX, USA.（德克萨斯大学达拉斯分校计算机科学系）；College of Computing, Georgia Institute of Technology, Atlanta, GA, USA（佐治亚理工学院计算机学院）；Department of Computer Science, University of Wisconsin-Madison, Madison, WI, USA（威斯康星大学麦迪逊分校计算机科学系）；Allen School of Computer Science, University of Washington, USA（华盛顿大学阿伦计算机科学学院）
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学穆斯克特基金会数据科学研究所，香港大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程系，香港大学）
Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Computer Science and Engineering, University of Texas, Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Massachusetts General Hospital, Harvard Medical School（麻省总医院哈佛医学院）；Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）
Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm
Key Laboratory of Pervasive Computing, Tsinghua University（清华大学普适计算重点实验室）；The University of New South Wales（新南威尔士大学）；SKLSDE Lab, Beihang University（北航SKLSDE实验室）
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学至理书院）
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
Brown University（布朗大学）；Columbia University（哥伦比亚大学）；Emory University（埃默里大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；UC San Diego（圣地亚哥大学）
AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models
National Yang Ming Chiao Tung University（国立阳明交通大学）；Institute of Computer Science and Engineering（工程与计算机科学学院）；College of Artificial Intelligence（人工智能学院）
针对VLMs理解空间语义时 allocentric 与 egocentric 参考框架的歧义问题，构建数据集AlloEgo-View并开发框架AlloEgo-VLM，经NVIDIA Isaac Sim平台验证其在具身机器人开放式物体搜索任务中的有效性。
From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation
Department of Engineering Science, University of Oxford（工程科学系，牛津大学）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Information Science and Technology, Northwest University（信息科学与技术学院，西北大学）
The University of Tokyo（东京大学）；Hunan University（湖南大学）
We present a new computational model for gaze prediction in egocentric videos by exploring patterns in temporal shift of gaze fixations (attention transition) that are dependent on egocentric manipulation tasks. Our assumption is that the high-level context of how a task is completed in a certain way has a strong influence on attention transition and should be modeled for gaze prediction in natural dynamic scenes. Specifically, we propose a hybrid model based on deep neural networks which integrates task-dependent attention transition with bottom-up saliency prediction. In particular, the task…
Egocentric perception has grown rapidly with the advent of immersive computing devices. Human gaze prediction is an important problem in analyzing egocentric videos and has primarily been tackled through either saliency-based modeling or highly supervised learning. We quantitatively analyze the generalization capabilities of supervised, deep learning models on the egocentric gaze prediction task on unseen, out-of-domain data. We find that their performance is highly dependent on the training data and is restricted to the domains specified in the training annotations. In this work, we tackle th…
This paper presents a framework for recognition of human activity from egocentric video and eye tracking data obtained from a head-mounted eye tracker. Three channels of information such as eye movement, ego-motion, and visual features are combined for the classification of activities. Image features were extracted using a pre-trained convolutional neural network. Eye and ego-motion are quantized, and the windowed histograms are used as the features. The combination of features obtains better accuracy for activity classification as compared to individual features.
It is well known that human gaze carries significant information about visual attention. However, there are three main difficulties in incorporating the gaze data in an attention mechanism of deep neural networks: 1) the gaze fixation points are likely to have measurement errors due to blinking and rapid eye movements; 2) it is unclear when and how much the gaze data is correlated with visual attention; and 3) gaze data is not always available in many real-world situations. In this work, we introduce an effective probabilistic approach to integrate human gaze into spatiotemporal attention for …
Universiti Teknologi Malaysia（马来西亚理工大学）；Universiti Malaysia Sarawak（马来西亚砂拉越大学）
Unsupervised segmentation of action segments in egocentric videos is a desirable feature in tasks such as activity recognition and content-based video retrieval. Reducing the search space into a finite set of action segments facilitates a faster and less noisy matching. However, there exist a substantial gap in machine understanding of natural temporal cuts during a continuous human activity. This work reports on a novel gaze-based approach for segmenting action segments in videos captured using an egocentric camera. Gaze is used to locate the region-of-interest inside a frame. By tracking two…
VLMs Guided Interpretable Decision Making for Autonomous Driving
Department of Computer Science, Tulane University（路易斯安那大学计算机科学系）；Qualcomm（高通公司）；Department of Industrial and Systems Engineering, North Carolina State University（北卡罗来纳州立大学工业与系统工程系）
