Gaze estimation for ordinary smart phone, e.g. estimating where the user is looking at on the phone screen, can be applied in various applications. However, the widely used appearance-based CNN methods still have two issues for practical adoption. First, due to the limited dataset, gaze estimation is very likely to suffer from over-fitting, leading to poor accuracy at run time. Second, the current methods are usually not robust, i.e. their prediction results having notable jitters even when the user is performing gaze fixation, which degrades user experience greatly. For the first issue, we propose a new tolerant and talented (TAT) training scheme, which is an iterative random knowledge distillation framework enhanced with cosine similarity pruning and aligned orthogonal initialization. The knowledge distillation is a tolerant teaching process providing diverse and informative supervision. The enhanced pruning and initialization is a talented learning process prompting the network to escape from the local minima and re-born from a better start. For the second issue, we define a new metric to measure the robustness of gaze estimator, and propose an adversarial training based Disturbance with Ordinal loss (DwO) method to improve it. The experimental results show that our TAT method achieves state-of-the-art performance on GazeCapture dataset, and that our DwO method improves the robustness while keeping comparable accuracy.
China University of Mining and Technology（中国矿业大学）；Australian National University（澳大利亚国立大学）
Vision based text entry systems aim to help disabled people achieve text communication using eye movement. Most previous methods have employed an existing eye tracker to predict gaze direction and design an input method based upon that. However, these methods can result in eye tracking quality becoming easily affected by various factors and lengthy amounts of time for calibration. Our paper presents a novel efficient gaze based text input method, which has the advantage of low cost and robustness. Users can type in words by looking at an on-screen keyboard and blinking. Rather than estimate ga…
University of Georgia（佐治亚大学）；Massachusetts Institute of Technology（麻省理工学院）；MPI Informatik（马克斯·普朗克信息学研究所）
From scientific research to commercial applications, eye tracking is an important tool across many domains. Despite its range of applications, eye tracking has yet to become a pervasive technology. We believe that we can put the power of eye tracking in everyone's palm by building eye tracking software that works on commodity hardware such as mobile phones and tablets, without the need for additional sensors or devices. We tackle this problem by introducing GazeCapture, the first large-scale dataset for eye tracking, containing data from over 1450 people consisting of almost 2.5M frames. Using…
Estimating correspondence between two images and extracting the foreground object are two challenges in computer vision. With dual-lens smart phones, such as iPhone 7Plus and Huawei P9, coming into the market, two images of slightly different views provide us new information to unify the two topics. We propose a joint method to tackle them simultaneously via a joint fully connected conditional random field (CRF) framework. The regional correspondence is used to handle textureless regions in matching and make our CRF system computationally efficient. Our method is evaluated over 2,000 new image…
National University of Sciences and Technology (NUST)（巴基斯坦国立科技大学（NUST））；Riphah International University（里法国际大学）
Unavailability of large training datasets is a bottleneck that needs to be overcome to realize the true potential of deep learning in histopathology applications. Although slide digitization via whole slide imaging scanners has increased the speed of data acquisition, labeling of virtual slides requires a substantial time investment from pathologists. Eye gaze annotations have the potential to speed up the slide labeling process. This work explores the viability and timing comparisons of eye gaze labeling compared to conventional manual labeling for training object detectors. Challenges associ…
Gaze and face tracking algorithms have traditionally battled a compromise between computational complexity and accuracy; the most accurate neural net algorithms cannot be implemented in real time, but less complex real-time algorithms suffer from higher error. This project seeks to better bridge that gap by improving on real-time eye and facial recognition algorithms in order to develop accurate, real-time gaze estimation with an emphasis on minimizing training data and computational complexity. Our goal is to use eye and facial recognition techniques to enable users to perform limited tasks b…
Homework grading is critical to evaluate teaching quality and effect. However, it is usually time-consuming to grade the homework manually. In automatic homework grading scenario, many optical mark reader (OMR)-based solutions which require specific equipments have been proposed. Although many of them can achieve relatively high accuracy, they are less convenient for users. In contrast, with the popularity of smart phones, the automatic grading system which depends on the image photographed by phones becomes more available. In practice, due to different photographing angles or uneven papers, i…
Kritikal Solutions Pvt. Ltd.（Kritikal Solutions 私人有限公司）；International Institute of Information Technology（国际信息技术学院）
Motion capturing and there by segmentation of the motion of any moving object from a sequence of continuous images or a video is not an exceptional task in computer vision area. Smart-phone camera application is an added integration for the development of such tasks and it also provides for a smooth testing. A new approach has been proposed for segmenting out the foreground moving object from the background and then masking the sequential motion with the static background which is commonly known as stroboscopic image. In this paper the whole process of the stroboscopic image construction techn…
Evaluating Sensitivity Parameters in Smartphone-Based Gaze Estimation: A Comparative Study of Appearance-Based and Infrared Eye Trackers
School of Computer, Data and Mathematical Sciences, Western Sydney University, Penrith, New South Wales, Australia（计算机、数据与数学科学学院，西悉尼大学，彭里，新南威尔士州，澳大利亚）；The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, Australia（脑行为与发育研究所，西悉尼大学，彭里，新南威尔士州，澳大利亚）；School of Systems and Computing, University of New South Wales, Canberra, ACT, Australia（系统与计算学院，新南威尔士大学，堪培拉，澳大利亚）
Zhejiang University of Technology（浙江工业大学）；Tianjin University of Technology（天津理工大学）
The efficiency and accuracy of mapping are crucial in a large scene and long-term AR applications. Multi-agent cooperative SLAM is the precondition of multi-user AR interaction. The cooperation of multiple smart phones has the potential to improve efficiency and robustness of task completion and can complete tasks that a single agent cannot do. However, it depends on robust communication, efficient location detection, robust mapping, and efficient information sharing among agents. We propose a multi-intelligence collaborative monocular visual-inertial SLAM deployed on multiple ios mobile devic…
Melanoma spreads through metastasis, and therefore it has been proven to be very fatal. Statistical evidence has revealed that the majority of deaths resulting from skin cancer are as a result of melanoma. Further investigations have shown that the survival rates in patients depend on the stage of the infection; early detection and intervention of melanoma implicates higher chances of cure. Clinical diagnosis and prognosis of melanoma is challenging since the processes are prone to misdiagnosis and inaccuracies due to doctors subjectivity. This paper proposes an innovative and fully functional…
Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm
Key Laboratory of Pervasive Computing, Tsinghua University（清华大学普适计算重点实验室）；The University of New South Wales（新南威尔士大学）；SKLSDE Lab, Beihang University（北航SKLSDE实验室）
Studies have shown that children can be exposed to smart devices at a very early age. This has important implications on research in children-computer interaction, children online safety and early education. Many systems have been built based on such research. In this work, we present multiple techniques to automatically detect the presence of a child on a smart device, which could be used as the first step on such systems. Our methods distinguish children from adults based on behavioral differences while operating a touch-enabled modern computing device. Behavioral differences are extracted f…
Recent research has demonstrated the ability to estimate gaze on mobile devices by performing inference on the image from the phone's front-facing camera, and without requiring specialized hardware. While this offers wide potential applications such as in human-computer interaction, medical diagnosis and accessibility (e.g., hands free gaze as input for patients with motor disorders), current methods are limited as they rely on collecting data from real users, which is a tedious and expensive process that is hard to scale across devices. There have been some attempts to synthesize eye region d…
Gaze-tracking is a novel way of interacting with computers which allows new scenarios, such as enabling people with motor-neuron disabilities to control their computers or doctors to interact with patient information without touching screen or keyboard. Further, there are emerging applications of gaze-tracking in interactive gaming, user experience research, human attention analysis and behavioral studies. Accurate estimation of the gaze may involve accounting for head-pose, head-position, eye rotation, distance from the object as well as operating conditions such as illumination, occlusion, b…
XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses
School of Software Engineering, Xi'an Jiaotong University（西安交通大学软件工程学院）；State Key Laboratory of Integrated Services Networks, Xidian University（西安电子科技大学信息网络服务国家重点实验室）；MOE KLINNS Lab, Xi'an Jiaotong University（教育部KLINNS实验室）；School of Computer Science and Technology, Xi'an Jiaotong University（西安交通大学计算机科学与技术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
