Deep neural networks for video-based eye tracking have demonstrated resilience to noisy environments, stray reflections, and low resolution. However, to train these networks, a large number of manually annotated images are required. To alleviate the cumbersome process of manual labeling, computer graphics rendering is employed to automatically generate a large corpus of annotated eye images under various conditions. In this work, we introduce a synthetic eye image generation platform that improves upon previous work by adding features such as an active deformable iris, an aspherical cornea, retinal retro-reflection, gaze-coordinated eye-lid deformations, and blinks. To demonstrate the utility of our platform, we render images reflecting the represented gaze distributions inherent in two publicly available datasets, NVGaze and OpenEDS. We also report on the performance of two semantic segmentation architectures (SegNet and RITnet) trained on rendered images and tested on the original datasets.
Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets. However, retaining relevant details from input data in the generated images is challenging and failure could be critical to the performance on the final task. In this work, we synthesize person-specific eye images that satisfy a given semantic segmentation mask (content), while following the style of a specified person from only a few reference images. We introduce two approaches, (a) one used to win the OpenEDS Synthetic Eye Generation Challenge at ICCV …
Accurate eye segmentation can improve eye-gaze estimation and support interactive computing based on visual attention; however, existing eye segmentation methods suffer from issues such as person-dependent accuracy, lack of robustness, and an inability to be run in real-time. Here, we present the RITnet model, which is a deep neural network that combines U-Net and DenseNet. RITnet is under 1 MB and achieves 95.3\% accuracy on the 2019 OpenEDS Semantic Segmentation challenge. Using a GeForce GTX 1080 Ti, RITnet tracks at $>$ 300Hz, enabling real-time gaze tracking applications. Pre-trained mode…
The overall goal of this work is to enrich training data for automated driving with so called corner cases. In road traffic, corner cases are critical, rare and unusual situations that challenge the perception by AI algorithms. For this purpose, we present the design of a test rig to generate synthetic corner cases using a human-in-the-loop approach. For the test rig, a real-time semantic segmentation network is trained and integrated into the driving simulation software CARLA in such a way that a human can drive on the network's prediction. In addition, a second person gets to see the same sc…
Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Recent studies reveal that deep neural network (DNN) based object detectors are vulnerable to adversarial attacks in the form of adding the perturbation to the images, leading to the wrong output of object detectors. Most current existing works focus on generating perturbed images, also called adversarial examples, to fool object detectors. Though the generated adversarial examples themselves can remain a certain naturalness, most of them can still be easily observed by human eyes, which limits their further application in the real world. To alleviate this problem, we propose a differential ev…
Universidade Federal do Espírito Santo (UFES)（圣埃斯皮里图联邦大学）；Instituto Federal do Espírito Santo (IFES)（圣埃斯皮里图联邦学院）
Modern lane detection methods have achieved remarkable performances in complex real-world scenarios, but many have issues maintaining real-time efficiency, which is important for autonomous vehicles. In this work, we propose LaneATT: an anchor-based deep lane detection model, which, akin to other generic deep object detectors, uses the anchors for the feature pooling step. Since lanes follow a regular pattern and are highly correlated, we hypothesize that in some cases global information may be crucial to infer their positions, especially in conditions such as occlusion, missing lane markers, …
Imperial College London（帝国理工学院）；Samsung AI Centre（三星人工智能中心）
Semantic segmentation of eyes has long been a vital pre-processing step in many biometric applications. Majority of the works focus only on high resolution eye images, while little has been done to segment the eyes from low quality images in the wild. However, this is a particularly interesting and meaningful topic, as eyes play a crucial role in conveying the emotional state and mental well-being of a person. In this work, we take two steps toward solving this problem: (1) We collect and annotate a challenging eye segmentation dataset containing 8882 eye patches from 4461 facial images of dif…
We presented an optical system to perform imaging interested objects in complex scenes, like the creature easy see the interested prey in the hunt for complex environments. It utilized Deep-learning network to learn the interested objects's vision features and designed the corresponding "imaging matrices", furthermore the learned matrixes act as the measurement matrix to complete compressive imaging with a single-pixel camera, finally we can using the compressed image data to only image the interested objects without the rest objects and backgrounds of the scenes with the previous Deep-learnin…
Ecole de technologie supérieure（高等技术学院）；Concordia University（康考迪亚大学）
The drone industry is diversifying and the number of pilots increases rapidly. In this context, flight schools need adapted tools to train pilots, most importantly with regard to their own awareness of their physiological and cognitive limits. In civil and military aviation, pilots can train themselves on realistic simulators to tune their reaction and reflexes, but also to gather data on their piloting behavior and physiological states. It helps them to improve their performances. Opposed to cockpit scenarios, drone teleoperation is conducted outdoor in the field, thus with only limited poten…
Research and Academic Computer Network（研究与学术计算机网络）；Warsaw University of Technology（华沙理工大学）；Medical University of Warsaw（华沙医科大学）
This paper presents an analysis of how the iris recognition is impacted by eye diseases and an appropriate dataset comprising 2996 iris images of 230 distinct eyes (including 184 illness-affected eyes representing more than 20 different eye conditions). The images were collected in near infrared and visible light during a routine ophthalmological practice. The experimental study shows four valuable results. First, the enrollment process is highly sensitive to those eye conditions that make the iris obstructed or introduce geometrical distortions. Second, even those conditions that do not produ…
In this work various methods and algorithms for face and eyes detection are examined in order to decide which of them are applicable for use in a driver fatigue monitoring system. In the case of face detection the standard Viola-Jones face detector has shown best results, while the method of finding the eye centers by means of gradients has proven to be most appropriate in the case of eyes detection. The later method has also a potential for retrieving behavioral parameters needed for estimation of the level of driver fatigue. This possibility will be examined in future work.
EyeSeg: An Uncertainty-Aware Eye Segmentation Framework for AR/VR
Shanghai Jiao Tong University（上海交通大学）；Tencent（腾讯）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；East China Normal University（华东师范大学）
Research and Academic Computer Network (NASK)（研究与学术计算机网络（NASK））；Warsaw University of Technology（华沙理工大学）；Medical University of Warsaw（华沙医科大学）
This paper presents a database of iris images collected from disease affected eyes and an analysis related to the influence of ocular diseases on iris recognition reliability. For that purpose we have collected a database of iris images acquired for 91 different eyes during routine ophthalmology visits. This collection gathers samples for healthy eyes as well as those with various eye pathologies, including cataract, acute glaucoma, posterior and anterior synechiae, retinal detachment, rubeosis iridis, corneal vascularization, corneal grafting, iris damage and atrophy and corneal ulcers, haze …
University of Cambridge（剑桥大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination condit…
A unique challenge in creating high-quality animatable and relightable 3D avatars of people is modeling human eyes. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to en…
Research and Academic Computer Network (NASK)（研究与学术计算机网络（NASK））；Warsaw University of Technology（华沙理工大学）；Medical University of Warsaw（华沙医科大学）
This paper presents an analysis of how iris recognition is influenced by eye disease and an appropriate dataset comprising 2996 images of irises taken from 230 distinct eyes (including 184 affected by more than 20 different eye conditions). The images were collected in near infrared and visible light during routine ophthalmological examination. The experimental study carried out utilizing four independent iris recognition algorithms (MIRLIN, VeriEye, OSIRIS and IriCore) renders four valuable results. First, the enrollment process is highly sensitive to those eye conditions that obstruct the ir…
A business's online, photographic representation can play a crucial role in its success or failure. We use Yelp's image dataset and star-based review system as a measurement of an image's effectiveness in promoting a business. After preprocessing the Yelp dataset, we use transfer learning to train a classifier which accepts Yelp images and predicts star-ratings. Additionally, we then train a GAN to qualitatively investigate the common properties of highly effective images. We achieve 90-98% accuracy in classifying simplified star ratings for various image categories and observe that images con…
We propose a novel method on refining cross-person gaze prediction task with eye/face images only by explicitly modelling the person-specific differences. Specifically, we first assume that we can obtain some initial gaze prediction results with existing method, which we refer to as InitNet, and then introduce three modules, the Validity Module (VM), Self-Calibration (SC) and Person-specific Transform (PT)) Module. By predicting the reliability of current eye/face images, our VM is able to identify invalid samples, e.g. eye blinking images, and reduce their effects in our modelling process. Ou…
Centre for Visual Information Technology, KCIS, IIIT Hyderabad（国际信息技术研究所（海得拉巴）KCIS 视觉信息技术中心）
Monocular head pose estimation requires learning a model that computes the intrinsic Euler angles for pose (yaw, pitch, roll) from an input image of human face. Annotating ground truth head pose angles for images in the wild is difficult and requires ad-hoc fitting procedures (which provides only coarse and approximate annotations). This highlights the need for approaches which can train on data captured in controlled environment and generalize on the images in the wild (with varying appearance and illumination of the face). Most present day deep learning approaches which learn a regression fu…
Vignan Institute of Technology and Sciences（维格纳科技学院）；VITS（维格纳科技学院）
Driving support systems, such as car navigation systems are becoming common and they support driver in several aspects. Non-intrusive method of detecting Fatigue and drowsiness based on eye-blink count and eye directed instruction controlhelps the driver to prevent from collision caused by drowsy driving. Eye detection and tracking under various conditions such as illumination, background, face alignment and facial expression makes the problem complex.Neural Network based algorithm is proposed in this paper to detect the eyes efficiently. In the proposed algorithm, first the neural Network is …
Gaze estimation has grown rapidly in accuracy in recent years. However, these models often fail to take advantage of different computer vision (CV) algorithms and techniques (such as small ResNet and Inception networks and ensemble models) that have been shown to improve results for other CV problems. Additionally, most current gaze estimation models require the use of either both eyes or an entire face, whereas real-world data may not always have both eyes in high resolution. Thus, we propose a gaze estimation model that implements the ResNet and Inception model architectures and makes predic…
Indian Institute of Technology Ropar（印度理工学院罗巴尔分校）；Monash University（莫纳什大学）
We present \textbf{FakeET}-- an eye-tracking database to understand human visual perception of \emph{deepfake} videos. Given that the principal purpose of deepfakes is to deceive human observers, FakeET is designed to understand and evaluate the ease with which viewers can detect synthetic video artifacts. FakeET contains viewing patterns compiled from 40 users via the \emph{Tobii} desktop eye-tracker for 811 videos from the \textit{Google Deepfake} dataset, with a minimum of two viewings per video. Additionally, EEG responses acquired via the \emph{Emotiv} sensor are also available. The compi…
EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment
State Key Laboratory of Integrated Services Networks, School of Electronic Engineering, Xidian University（信息服务网络国家重点实验室，电子工程学院，西安电子科技大学）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University（协同智能系统重点实验室，教育部，西安电子科技大学）；College of Mathematical Science, Inner Mongolia Normal University（数学科学学院，内蒙古师范大学）；School of Electronic Engineering, Xidian University（电子工程学院，西安电子科技大学）
Indian Institute of Technology Hyderabad（印度海得拉巴理工学院）；University of Pittsburgh（匹兹堡大学）；L V Prasad Eye Institute（L V 普拉萨德眼科研究所）
A variety of vision ailments are associated with geographic atrophy (GA) in the foveal region of the eye. In current clinical practice, the ophthalmologist manually detects potential presence of such GA based on fundus autofluorescence (FAF) images, and hence diagnoses the disease, when relevant. However, in view of the general scarcity of ophthalmologists relative to the large number of subjects seeking eyecare, especially in remote regions, it becomes imperative to develop methods to direct expert time and effort to medically significant cases. Further, subjects from either disadvantaged bac…
Pedestrians are particularly vulnerable road users in urban traffic. With the arrival of autonomous driving, novel technologies can be developed specifically to protect pedestrians. We propose a machine learning toolchain to train artificial neural networks as models of pedestrian behavior. In a preliminary study, we use synthetic data from simulations of a specific pedestrian crossing scenario to train a variational autoencoder and a long short-term memory network to predict a pedestrian's future visual perception. We can accurately predict a pedestrian's future perceptions within relevant ti…
Facebook Reality Labs（Facebook现实实验室）；Facebook；Texas State University（德克萨斯州立大学）；Universitat de Barcelona（巴塞罗那大学）
We present the second edition of OpenEDS dataset, OpenEDS2020, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display mounted with two synchronized eye-facing cameras. The dataset, which is anonymized to remove any personally identifiable information on participants, consists of 80 participants of varied appearance performing several gaze-elicited tasks, and is divided in two subsets: 1) Gaze Prediction Dataset, with up to 66,560 sequences containing 550,400 eye-images and respective gaze vectors, cr…
College of Engineering Roorkee（鲁尔基工程学院）；Indraprastha Institute of Information Technology Delhi (IIIT-Delhi)（德里英德拉普拉斯塔信息技术学院（IIIT-德里））；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Raman Classes（拉曼课堂）
In this paper, we present how Bell's Palsy, a neurological disorder, can be detected just from a subject's eyes in a video. We notice that Bell's Palsy patients often struggle to blink their eyes on the affected side. As a result, we can observe a clear contrast between the blinking patterns of the two eyes. Although previous works did utilize images/videos to detect this disorder, none have explicitly focused on the eyes. Most of them require the entire face. One obvious advantage of having an eye-focused detection system is that subjects' anonymity is not at risk. Also, our AI decisions base…
What if Eye...? Computationally Recreating Vision Evolution
Camera Culture；MIT Media Lab（麻省理工学院媒体实验室）；Computational Imaging Lab（计算成像实验室）；Rice University（莱斯大学）；Center for Brains Minds and Machines（大脑、心智与机器中心）；MIT（麻省理工学院）；Lund Vision Group（隆德视觉组）；Lund University（隆德大学）；InfoLab（信息实验室）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Charles Stark Draper Laboratory（查尔斯·斯塔克·德雷珀实验室）
Open Your Eyes: Vision Enhances Message Passing Neural Networks in Link Prediction
Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）；Zhejiang University（浙江大学）
