Gaze estimation is a fundamental task in many applications of computer vision, human computer interaction and robotics. Many state-of-the-art methods are trained and tested on custom datasets, making comparison across methods challenging. Furthermore, existing gaze estimation datasets have limited head pose and gaze variations, and the evaluations are conducted using different protocols and metrics. In this paper, we propose a new gaze estimation dataset called ETH-XGaze, consisting of over one million high-resolution images of varying gaze under extreme head poses. We collect this dataset from 110 participants with a custom hardware setup including 18 digital SLR cameras and adjustable illumination conditions, and a calibrated system to record ground truth gaze targets. We show that our dataset can significantly improve the robustness of gaze estimation methods across different head poses and gaze angles. Additionally, we define a standardized experimental protocol and evaluation metric on ETH-XGaze, to better unify gaze estimation research going forward. The dataset and benchmark website are available at https://ait.ethz.ch/projects/2020/ETH-XGaze
Gaze estimation involves predicting where the person is looking at within an image or video. Technically, the gaze information can be inferred from two different magnification levels: face orientation and eye orientation. The inference is not always feasible for gaze estimation in the wild, given the lack of clear eye patches in conditions like extreme left/right gazes or occlusions. In this work, we design a model that mimics humans' ability to estimate the gaze by aggregating from focused looks, each at a different magnification level of the face area. The model avoids the need to extract cl…
We propose a novel method on refining cross-person gaze prediction task with eye/face images only by explicitly modelling the person-specific differences. Specifically, we first assume that we can obtain some initial gaze prediction results with existing method, which we refer to as InitNet, and then introduce three modules, the Validity Module (VM), Self-Calibration (SC) and Person-specific Transform (PT)) Module. By predicting the reliability of current eye/face images, our VM is able to identify invalid samples, e.g. eye blinking images, and reduce their effects in our modelling process. Ou…
Tomorrow Advancing Life (TAL) Education Group（好未来（TAL）教育集团）
This paper presents a method for gaze estimation according to face images. We train several gaze estimators adopting four different network architectures, including an architecture designed for gaze estimation (i.e.,iTracker-MHSA) and three originally designed for general computer vision tasks(i.e., BoTNet, HRNet, ResNeSt). Then, we select the best six estimators and ensemble their predictions through a linear combination. The method ranks the first on the leader-board of ETH-XGaze Competition, achieving an average angular error of $3.11^{\circ}$ on the ETH-XGaze test set.
Is Geometry Enough? An Evaluation of Landmark-Based Gaze Estimation
Department of Industrial Engineering and Mathematical Sciences, Università Politecnica delle Marche（工业工程与数学科学系，帕尔米塞大学）；Department of Science and Information Technology, Università Pegaso（科学与信息科技系，佩加索大学）
Deep neural networks have significantly improved appearance-based gaze estimation accuracy. However, it still suffers from unsatisfactory performance when generalizing the trained model to new domains, e.g., unseen environments or persons. In this paper, we propose a plug-and-play gaze adaptation framework (PnP-GA), which is an ensemble of networks that learn collaboratively with the guidance of outliers. Since our proposed framework does not require ground-truth labels in the target domain, the existing gaze estimation networks can be directly plugged into PnP-GA and generalize the algorithms…
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China（桂林电子科技大学人工智能学院）；The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada（图像、视觉与人工智能实验室，蒙特利尔大学高级技术学院）；Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia（计算机与信息科学学院信息科技系，沙特国王大学）；Department of Cybersecurity, College of Computer Science and Engineering, Taibah University, Medina, Saudi Arabia（计算机科学与工程学院网络安全系，塔伊卜大学）；Department of Networked Engineering, College of Computer Science and Engineering, Jeddah University, Jeddah, Saudi Arabia（计算机科学与工程学院网络工程系，吉达大学）
Monash University（莫纳什大学）；Indian Institute of Technology Ropar（罗帕尔印度理工学院）；Kroop AI；University of Trento（特伦托大学）
Labelling of human behavior analysis data is a complex and time consuming task. In this paper, a fully automatic technique for labelling an image based gaze behavior dataset for driver gaze zone estimation is proposed. Domain knowledge is added to the data recording paradigm and later labels are generated in an automatic manner using Speech To Text conversion (STT). In order to remove the noise in the STT process due to different illumination and ethnicity of subjects in our data, the speech frequency and energy are analysed. The resultant Driver Gaze in the Wild (DGW) dataset contains 586 rec…
Estimating eye-gaze from images alone is a challenging task, in large parts due to un-observable person-specific factors. Achieving high accuracy typically requires labeled data from test users which may not be attainable in real applications. We observe that there exists a strong relationship between what users are looking at and the appearance of the user's eyes. In response to this understanding, we propose a novel dataset and accompanying method which aims to explicitly learn these semantic and temporal relationships. Our video dataset consists of time-synchronized screen recordings, user-…
Sapienza University of Rome（罗马第一大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center (CVC)（计算机视觉中心）
Recent joint embedding-based self-supervised methods have surpassed standard supervised approaches on various image recognition tasks such as image classification. These self-supervised methods aim at maximizing agreement between features extracted from two differently transformed views of the same image, which results in learning an invariant representation with respect to appearance and geometric image transformations. However, the effectiveness of these approaches remains unclear in the context of gaze estimation, a structured regression task that requires equivariance under geometric trans…
University of Minnesota（明尼苏达大学）；POSTECH（浦项科技大学）；BinaryVR
This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and physical condition. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance using their canonical atlas. We demonstrate that HUMBI is highl…
Coordinate Singularities Break Conformal Coverage for Gaze and Head Pose
Smart Health Technologies Group, Centre for Computational Intelligence, School of Computer Science and Electronic Engineering, University of Essex（智能健康技术组，计算智能中心，埃塞克斯大学计算机科学与电子工程学院）
This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. The goal of HUMBI is to facilitate modeling view-specific appearance and geometry of five primary body signals including gaze, face, hand, body, and garment from assorted people. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and style. With the multiview image streams, we reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance. We demonstrate that HUMBI is highly eff…
GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
AIPM, School of Artificial Intelligence, Guilin University of Electronic Technology, China（AIPM，人工智能学院，桂林电子科技大学，中国）；Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Canada（影像、视觉与人工智能实验室，École de Technologie Supérieure，加拿大）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
We present labelled pupils in the wild (LPW), a novel dataset of 66 high-quality, high-speed eye region videos for the development and evaluation of pupil detection algorithms. The videos in our dataset were recorded from 22 participants in everyday locations at about 95 FPS using a state-of-the-art dark-pupil head-mounted eye tracker. They cover people with different ethnicities, a diverse set of everyday indoor and outdoor illumination environments, as well as natural gaze direction distributions. The dataset also includes participants wearing glasses, contact lenses, as well as make-up. We …
Fundación Vicomtech（维科姆泰克基金会）；Basque Research and Technology Alliance（巴斯克研究与技术联盟）；Clemson University（克莱姆森大学）；University of the Basque Country（巴斯克大学）；Ikerbasque（伊克巴斯克基金会）；Donostia International Physics Center（圣塞瓦斯蒂安国际物理中心）；Biofisika Institute（生物物理研究所）；Harvard John A. Paulson School of Engineering and Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Boston College（波士顿学院）
The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video
Barnard College, Columbia University（哥伦比亚大学巴纳德学院）；Bates College（贝茨学院）；North Dakota State University（北达科他州立大学）；University of Nevada, Reno（内华达大学里诺分校）；Magic Leap；Technical University of Munich（慕尼黑工业大学）；Unmanned Ground Systems（无人地面系统公司）；Smith-Kettlewell Eye Research Institute（史密斯-凯特威尔眼科研究所）
Many computer vision tasks rely on labeled data. Rapid progress in generative modeling has led to the ability to synthesize photorealistic images. However, controlling specific aspects of the generation process such that the data can be used for supervision of downstream tasks remains challenging. In this paper we propose a novel generative model for images of faces, that is capable of producing high-quality images under fine-grained control over eye gaze and head orientation angles. This requires the disentangling of many appearance related factors including gaze and head orientation but also…
Facebook AI Research (FAIR)（脸书人工智能研究院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Georgia Tech（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）；University of Minnesota（明尼苏达大学）；Facebook（脸书）；University of Bristol（布里斯托尔大学）；Carnegie Mellon University Africa（卡内基梅隆大学非洲分校）；Universidad de los Andes（安第斯大学）；Facebook Reality Labs（脸书现实实验室）；University of Tokyo（东京大学）；Indiana University（印第安纳大学）
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Port…
