Eye Gaze Controlled Robotic Arm for Persons with SSMI
Indian Institute of Science（印度科学学院）；Indian Institute of Technology, Madras（马德拉斯印度理工学院）；Indian Institute of Information Technology, Kalyani（卡利亚尼印度信息技术学院）
Vinay Krishna Sharma, L. R. D. Murthy, KamalPreet Singh Saluja, Vimal Mollyn, Gourav Sharma, Pradipta Biswas
备注：
Citation: VK Sharma, KPS Saluja, LRD Murthy, G Sharma and P Biswas, Webcam Controlled Robotic Arm for Persons with SSMI, Technology and Disability 32 (3), IOS Press 2020 [Official journal of EU AAATE association]
Background: People with severe speech and motor impairment (SSMI) often uses a technique called eye pointing to communicate with outside world. One of their parents, caretakers or teachers hold a printed board in front of them and by analyzing their eye gaze manually, their intentions are interpreted. This technique is often error prone and time consuming and depends on a single caretaker.
Objective: We aimed to automate the eye tracking process electronically by using commercially available tablet, computer or laptop and without requiring any dedicated hardware for eye gaze tracking. The eye gaze tracker is used to develop a video see through based AR (augmented reality) display that controls a robotic device with eye gaze and deployed for a fabric printing task.
Methodology: We undertook a user centred design process and separately evaluated the web cam based gaze tracker and the video see through based human robot interaction involving users with SSMI. We also reported a user study on manipulating a robotic arm with webcam based eye gaze tracker.
Results: Using our bespoke eye gaze controlled interface, able bodied users can select one of nine regions of screen at a median of less than 2 secs and users with SSMI can do so at a median of 4 secs. Using the eye gaze controlled human-robot AR display, users with SSMI could undertake representative pick and drop task at an average duration less than 15 secs and reach a randomly designated target within 60 secs using a COTS eye tracker and at an average time of 2 mins using the webcam based eye gaze tracker.
The Hong Kong University of Science and Technology（香港科技大学）
Vision-based autonomous driving through imitation learning mimics the behaviors of human drivers by training on pairs of data of raw driver-view images and actions. However, there are other cues, e.g. gaze behavior, available from human drivers that have yet to be exploited. Previous research has shown that novice human learners can benefit from observing experts' gaze patterns. We show here that deep neural networks can also benefit from this. We demonstrate different approaches to integrating gaze information into imitation networks. Our results show that the integration of gaze information …
Technological Institute of the Philippines（菲律宾理工学院）
Eye-GUIDE is an assistive communication tool designed for the paralyzed or physically impaired people who were unable to move parts of their bodies especially people whose communications are limited only to eye movements. The prototype consists of a camera and a computer. Camera captures images then it will be send to the computer, where the computer will be the one to interpret the data. Thus, Eye-GUIDE focuses on camera-based gaze tracking. The proponent designed the prototype to perform simple tasks and provides graphical user interface in order the paralyzed or physically impaired person c…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
3D gaze information is important for scene-centric attention analysis but accurate estimation and analysis of 3D gaze in real-world environments remains challenging. We present a novel 3D gaze estimation method for monocular head-mounted eye trackers. In contrast to previous work, our method does not aim to infer 3D eyeball poses but directly maps 2D pupil positions to 3D gaze directions in scene camera coordinate space. We first provide a detailed discussion of the 3D gaze estimation task and summarize different methods, including our own. We then evaluate the performance of different 3D gaze…
University of Stuttgart（斯图加特大学）；University of Southampton（南安普顿大学）
World-wide-web, with the website and webpage as the main interface, facilitates the dissemination of important information. Hence it is crucial to optimize them for better user interaction, which is primarily done by analyzing users' behavior, especially users' eye-gaze locations. However, gathering these data is still considered to be labor and time intensive. In this work, we enable the development of automatic eye-gaze estimations given a website screenshots as the input. This is done by the curation of a unified dataset that consists of website screenshots, eye-gaze heatmap and website's l…
University of Science and Technology of China（中国科学技术大学）；Carnegie Mellon University（卡内基梅隆大学）
In recent years, head-mounted near-eye display devices have become the key hardware foundation for virtual reality and augmented reality. Thus head-mounted gaze tracking technology has received attention as an essential part of human-computer interaction. However, unavoidable slippage of head-mounted devices (HMD) often results higher gaze tracking errors and hinders the practical usage of HMD. To tackle this problem, we propose a slippage-robust gaze tracking for near-eye display method based on the aspheric eyeball model and accurately compute the eyeball optical axis and rotation center. We…
Despite a revolution in the pervasiveness of video cameras in our daily lives, one of the most meaningful forms of nonverbal affective communication, interpersonal eye gaze, i.e. eye gaze relative to a conversation partner, is not available from common video. We introduce the Interpersonal-Calibrating Eye-gaze Encoder (ICE), which automatically extracts interpersonal gaze from video recordings without specialized hardware and without prior knowledge of participant locations. Leveraging the intuition that individuals spend a large portion of a conversation looking at each other enables the ICE …
Pupil Labs UG（Pupil Labs公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software. This limits the application areas and use of mobile eye tracking to expert users and inhibits user-driven development, customisation, and extension. In this paper we present Pupil -- an accessible, affordable, and extensible open source platform for mobile eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight headset with high-resolution cameras, 2) an open source software framework for mobile eye tracking, as…
With the emergence of Virtual and Mixed Reality (XR) devices, eye tracking has received significant attention in the computer vision community. Eye gaze estimation is a crucial component in XR -- enabling energy efficient rendering, multi-focal displays, and effective interaction with content. In head-mounted XR devices, the eyes are imaged off-axis to avoid blocking the field of view. This leads to increased challenges in inferring eye related quantities and simultaneously provides an opportunity to develop accurate and robust learning based approaches. To this end, we present MagicEyes, the …
Eye gaze controlled interfaces allow us to directly manipulate a graphical user interface just by looking at it. This technology has great potential in military aviation, in particular, operating different displays in situations where pilots hands are occupied with flying the aircraft. This paper reports studies on analyzing accuracy of eye gaze controlled interface inside aircraft undertaking representative flying missions. We reported that pilots can undertake representative pointing and selection tasks at less than 2 secs on average. Further, we evaluated the accuracy of eye gaze tracking g…
Massachusetts Institute of Technology (MIT)（麻省理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；SAFER
Accurate, robust, inexpensive gaze tracking in the car can help keep a driver safe by facilitating the more effective study of how to improve (1) vehicle interfaces and (2) the design of future Advanced Driver Assistance Systems. In this paper, we estimate head pose and eye pose from monocular video using methods developed extensively in prior work and ask two new interesting questions. First, how much better can we classify driver gaze using head and eye pose versus just using head pose? Second, are there individual-specific gaze strategies that strongly correlate with how much gaze classific…
Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
Hearing-impaired listeners usually have troubles attending target talker in multi-talker scenes, even with hearing aids (HAs). The problem can be solved with eye-gaze steering HAs, which requires listeners eye-gazing on the target. In a situation where head rotates, eye-gaze is subject to both behaviors of saccade and head rotation. However, existing methods of eye-gaze estimation did not work reliably, since the listener's strategy of eye-gaze varies and measurements of the two behaviors were not properly combined. Besides, existing methods were based on hand-craft features, which could overl…
Analyzing the gaze accuracy characteristics of an eye tracker is a critical task as its gaze data is frequently affected by non-ideal operating conditions in various consumer eye tracking applications. In this study, gaze error patterns produced by a commercial eye tracking device were studied with the help of machine learning algorithms, such as classifiers and regression models. Gaze data were collected from a group of participants under multiple conditions that commonly affect eye trackers operating on desktop and handheld platforms. These conditions (referred here as error sources) include…
In this paper, we consider the problem of tracking the eye-gaze of individuals while they engage in reading. Particularly, we develop ways to accurately track the line being read by an individual using commercially available eye tracking devices. Such an approach will enable futuristic functionalities such as comprehension evaluation, interest level detection, and user-assisting applications like hands-free navigation and automatic scrolling. Existing commercial eye trackers provide an estimated location of the eye-gaze fixations every few milliseconds. However, this estimated data is found to…
University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers to perform, e.g., low latency predictive rendering, or to study quick and subtle eye motions like microsaccades using head-mounted devices in the wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 Hz with an accuracy that matches that of high-end desktop-mounted commercial trackers when evaluated in the same conditions. Our s…
INESC-ID（INESC-ID（葡萄牙国家系统与计算机工程研究所））；Instituto Superior Técnico（里斯本高等技术学院）；Universidade de Lisboa（里斯本大学）；School of Information Systems（信息系统学院）；Queensland University of Technology（昆士兰科技大学）；Grupo Lusíadas（卢西亚达斯集团）
This paper proposes a novel multimodal DL architecture incorporating medical images and eye-tracking data for abnormality detection in chest x-rays. Our results show that applying eye gaze data directly into DL architectures does not show superior predictive performance in abnormality detection chest X-rays. These results support other works in the literature and suggest that human-generated data, such as eye gaze, needs a more thorough investigation before being applied to DL architectures.
In this paper a review is presented of the research on eye gaze estimation techniques and applications, that has progressed in diverse ways over the past two decades. Several generic eye gaze use-cases are identified: desktop, TV, head-mounted, automotive and handheld devices. Analysis of the literature leads to the identification of several platform specific factors that influence gaze tracking accuracy. A key outcome from this review is the realization of a need to develop standardized methodologies for performance evaluation of gaze tracking systems and achieve consistency in their specific…
Gwangju Institute of Science and Technology（光州科学技术院）
Human-Computer Interaction(HCI) is a field that studies interactions between human users and computer systems. With the development of HCI, individuals or groups of people can use various digital technologies to achieve the optimal user experience. Human visual attention and visual intelligence are related to cognitive science, psychology, and marketing informatics, and are used in various applications of HCI. Gaze recognition is closely related to the HCI field because it is meaningful in that it can enhance understanding of basic human behavior. We can obtain reliable visual attention by the…
In this paper, we present an analysis of eye gaze patterns pertaining to visual cues in augmented reality (AR) for head-mounted displays (HMDs). We conducted an experimental study involving a picking and assembly task, which was guided by different visual cues. We compare these visual cues along multiple dimensions (in-view vs. out-of-view, static vs. dynamic, sequential vs. simultaneous) and analyze quantitative metrics such as gaze distribution, gaze duration, and gaze path distance. Our results indicate that visual cues in AR significantly affect eye gaze patterns. Specifically, we show tha…
Eye movements hold information about human perception, intention, and cognitive state. We propose a novel eye movement simulator that i) probabilistically simulates saccade movements as gamma distributions considering different peak velocities and ii) models smooth pursuit onsets with the sigmoid function. Additionally, it is capable of producing velocity and two-dimensional gaze sequences for static and dynamic scenes using saliency maps or real fixation targets. Our approach is also capable of simulating any sampling rate, even with uctuations. The simulation is evaluated against publicly av…
Gaze tracking is increasingly becoming an essential component in Augmented and Virtual Reality. Modern gaze tracking al gorithms are heavyweight; they operate at most 5 Hz on mobile processors despite that near-eye cameras comfortably operate at a r eal-time rate ($>$ 30 Hz). This paper presents a real-time eye tracking algorithm that, on average, operates at 30 Hz on a mobile processor, achieves \ang{0.1}--\ang{0.5} gaze accuracies, all the while requiring only 30K parameters, one to two orders of magn itude smaller than state-of-the-art eye tracking algorithms. The crux of our algorithm is a…
Technical progress in hardware and software enables us to record gaze data in everyday situations and over long time spans. Among a multitude of research opportunities, this technology enables visualization researchers to catch a glimpse behind performance measures and into the perceptual and cognitive processes of people using visualization techniques. The majority of eye tracking studies performed for visualization research is limited to the analysis of gaze distributions and aggregated statistics, thus only covering a small portion of insights that can be derived from gaze data. We argue th…
In recent times, there has been significant interest in the machine recognition of human emotions, due to the suite of applications to which this knowledge can be applied. A number of different modalities, such as speech or facial expression, individually and with eye gaze, have been investigated by the affective computing research community to either classify the emotion (e.g. sad, happy, angry) or predict the continuous values of affective dimensions (e.g. valence, arousal, dominance) at each moment in time. Surprisingly after an extensive literature review, eye gaze as a unimodal input to a…
Conducting collaborative tasks, e.g., multi-user game, in virtual reality (VR) could enable us to explore more immersive and effective experience. However, for current VR systems, users cannot communicate properly with each other via their gaze points, and this would interfere with users' mutual understanding of the intention. In this study, we aimed to find the optimal eye tracking data visualization , which minimized the cognitive interference and improved the understanding of the visual attention and intention between users. We designed three different eye tracking data visualizations: gaze…
In this paper, we propose an approach to track the progression of eye-gaze while reading a block of text on computer screen. The proposed approach will help to accurately quantify reading, e.g., identifying the lines of text that were read/skipped and estimating the time spent on each line, based on commercially available inexpensive eye-tracking devices. The proposed approach is based on a novel Slip Kalman filter that is custom designed to track the progression of reading. The performance of the proposed method is demonstrated using 25 pages eye-tracking data collected using a commercial des…
Webcam eye tracking for the collection of gaze data in the context of user studies is convenient - it can be used in remote tests where participants do not need special hardware. The approach has strong limitations, especially regarding the motion-free nature of the test persons during data recording and the quality of the gaze data obtained. Our study with 52 participants shows that usable eye tracking data can be obtained with commercially available webcams in a remote setting. However, a high drop off rate must be considered, which is why we recommend a high over-recruitment of 150%. We als…
While being able to read with screen magnifiers, low vision people have slow and unpleasant reading experiences. Eye tracking has the potential to improve their experience by recognizing fine-grained gaze behaviors and providing more targeted enhancements. To inspire gaze-based low vision technology, we investigate the suitable method to collect low vision users' gaze data via commercial eye trackers and thoroughly explore their challenges in reading based on their gaze behaviors. With an improved calibration interface, we collected the gaze data of 20 low vision participants and 20 sighted co…
In this paper we consider the problem of tracking the progression of reading through eye-gaze measurements. Such an algorithm is novel and will ultimately help to develop a method of analyzing eye-gaze data which had been collected during reading activity in order to uncover crucial information regarding the individual's interest level and quality of experience while reading a passage of text or book. Additionally, such an approach will serve as a "visual signature" - a method of verifying if an individual has indeed given adequate attention to critical text-based information. Further, an accu…
Speech has been a widely used modality in the field of affective computing. Recently however, there has been a growing interest in the use of multi-modal affective computing systems. These multi-modal systems incorporate both verbal and non-verbal features for affective computing tasks. Such multi-modal affective computing systems are advantageous for emotion assessment of individuals in audio-video communication environments such as teleconferencing, healthcare, and education. From a review of the literature, the use of eye gaze features extracted from video is a modality that has remained la…
In this paper, we firstly present what is Interactive Evolutionary Computation (IEC) and rapidly how we have combined this artificial intelligence technique with an eye-tracker for visual optimization. Next, in order to correctly parameterize our application, we present results from applying data mining techniques on gaze information coming from experiments conducted on about 80 human individuals.
We present GAZED- eye GAZe-guided EDiting for videos captured by a solitary, static, wide-angle and high-resolution camera. Eye-gaze has been effectively employed in computational applications as a cue to capture interesting scene content; we employ gaze as a proxy to select shots for inclusion in the edited video. Given the original video, scene content and user eye-gaze tracks are combined to generate an edited video comprising cinematically valid actor shots and shot transitions to generate an aesthetic and vivid representation of the original narrative. We model cinematic video editing as …
Imperial College London（帝国理工学院）；Data Science Institute（数据科学研究所）
Eye movements have long been studied as a window into the attentional mechanisms of the human brain and made accessible as novelty style human-machine interfaces. However, not everything that we gaze upon, is something we want to interact with; this is known as the Midas Touch problem for gaze interfaces. To overcome the Midas Touch problem, present interfaces tend not to rely on natural gaze cues, but rather use dwell time or gaze gestures. Here we present an entirely data-driven approach to decode human intention for object manipulation tasks based solely on natural gaze cues. We run data co…
Deep Learning-Based Visual Fatigue Detection Using Eye Gaze Patterns in VR
Dept. of Computer Science Clarkson University Potsdam, New York, USA（计算机科学系 克拉克逊大学）；David D. Reh School of Business Clarkson University Potsdam, New York, USA（戴维·D·雷商学院 克拉克逊大学）
Islamic University of Technology（伊斯兰技术大学）；University of Manitoba（曼尼托巴大学）；National University, Bangladesh（孟加拉国国立大学）
Human intention is an internal, mental characterization for acquiring desired information. From interactive interfaces containing either textual or graphical information, intention to perceive desired information is subjective and strongly connected with eye gaze. In this work, we determine such intention by analyzing real-time eye gaze data with a low-cost regular webcam. We extracted unique features (e.g., Fixation Count, Eye Movement Ratio) from the eye gaze data of 31 participants to generate a dataset containing 124 samples of visual intention for perceiving textual or graphical informati…
One of the main challenges of gaze-based interactions is the ability to distinguish normal eye function from a deliberate interaction with the computer system, commonly referred to as 'Midas touch'. In this paper we propose, EyeTAP (Eye tracking point-and-select by Targeted Acoustic Pulse) a hands-free interaction method for point-and-select tasks. We evaluated the prototype in two separate user studies, each containing two experiments with 33 participants and found that EyeTAP is robust even in presence of ambient noise in the audio input signal with tolerance of up to 70 dB, results in a fas…
In this paper, we describe the results of replicating one of our studies from two years ago which compares two techniques for transferring content across multiple screens in VR. Results from the previous study have shown that a combined gaze and touch input can outperform a bimanual touch-only input in terms of task completion time, simulator sickness, task load and usability. Except for the simulator sickness, these findings could be validated by the replication. The difference with regards to simulator sickness and variations in absolute scores of the other measures could be explained by a d…
Indian Institute of Technology（印度理工学院）；Indian Institute of Information Technology（印度信息技术学院）
A user's eyes provide means for Human Computer Interaction (HCI) research as an important modal. The time to time scientific explorations of the eye has already seen an upsurge of the benefits in HCI applications from gaze estimation to the measure of attentiveness of a user looking at a screen for a given time period. The eye tracking system as an assisting, interactive tool can be incorporated by physically disabled individuals, fitted best for those who have eyes as only a limited set of communication. The threefold objective of this paper is - 1. To introduce a neural network based archite…
