Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University(虚拟现实技术与系统国家重点实验室,北京航空航天大学计算机科学与工程学院);College of AI, Tsinghua University(清华大学人工智能学院)
Speak2Label: Using Domain Knowledge for Creating a Large Scale Driver Gaze Zone Estimation Dataset
Monash University(莫纳什大学);Indian Institute of Technology Ropar(罗帕尔印度理工学院);Kroop AI;University of Trento(特伦托大学)
人类行为分析数据的标记是一项复杂且耗时的任务。在本文中,提出了一种全自动的技术标记基于图像的凝视行为数据集的驾驶员凝视区估计。领域知识被添加到数据记录范例中,随后使用语音到文本转换(STT)以自动方式生成标签。为了消除STT过程中由于数据中对象的照明和种族不同而产生的噪音,对语音频率和能量进行了分析。由此产生的Driver Gaze in the Wild(DGW)数据集包含586个记录,这些记录是在一天中的不同时间(包括晚上)捕获的。大规模数据集包含338名年龄范围为18-63岁的受试者。由于数据是在不同的光照条件下记录的,因此在卷积神经网络(CNN)中提出了光照鲁棒层。大量的实验表明,数据集的方差类似于现实世界的条件和建议的CNN管道的有效性。所提出的网络还针对眼睛注视预测任务进行了微调,这显示了我们的网络在所提出的DGW数据集上学习的表示的区分性。项目页面:https://sites.google.com/view/drivergazeprediction/home
Northwestern Polytechnical University(西北工业大学);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);Athens Academy(雅典学院);ShanghaiTech University(上海科技大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校)