Large-scale public datasets have been shown to benefit research in multiple areas of modern artificial intelligence. For decision-making research that requires human data, high-quality datasets serve as important benchmarks to facilitate the development of new methods by providing a common reproducible standard. Many human decision-making tasks require visual attention to obtain high levels of performance. Therefore, measuring eye movements can provide a rich source of information about the strategies that humans use to solve decision-making tasks. Here, we provide a large-scale, high-quality dataset of human actions with simultaneously recorded eye movements while humans play Atari video games. The dataset consists of 117 hours of gameplay data from a diverse set of 20 games, with 8 million action demonstrations and 328 million gaze samples. We introduce a novel form of gameplay, in which the human plays in a semi-frame-by-frame manner. This leads to near-optimal game decisions and game scores that are comparable or better than known human records. We demonstrate the usefulness of the dataset through two simple applications: predicting human gaze and imitating human demonstrated actions. The quality of the data leads to promising results in both tasks. Moreover, using a learned human gaze model to inform imitation learning leads to an 115\% increase in game performance. We interpret these results as highlighting the importance of incorporating human visual attention in models of decision making and demonstrating the value of the current dataset to the research community. We hope that the scale and quality of this dataset can provide more opportunities to researchers in the areas of visual attention, imitation learning, and reinforcement learning.
Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities
Golisano College of Computing and Information Sciences, RIT(罗切斯特理工学院戈利萨诺计算与信息科学学院);Chester F. Carlson Center for Imaging Science, RIT(罗切斯特理工学院切斯特·F·卡尔森成像科学中心)
TalkingEyes: Pluralistic Speech-Driven 3D Eye Gaze Animation
School of Informatics, Xiamen University(厦门大学信息学院);China Mobile (Hangzhou) Information Technology Co., Ltd.(中国移动(杭州)信息技术有限公司);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
Estimating Central, Peripheral, and Temporal Visual Contributions to Human Decision Making in Atari Games
Department of Advanced Interdisciplinary Studies, The University of Tokyo(东京大学先进跨学科研究系);Research Center for Advanced Science and Technology, The University of Tokyo(东京大学先进科学与技术研究中心)
Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning
University of Massachusetts Boston(马萨诸塞大学波士顿分校);Kathmandu University(加德满都大学);Wentworth Institute of Technology(温特沃斯理工学院);Washington State University(华盛顿州立大学);University of Trás-os-Montes e Alto Douro(特拉斯奥斯蒙特斯和上杜罗大学);Institute of Electronics and Informatics Engineering of Aveiro(阿威罗电子与信息学工程研究所);INESC TEC
Eye Gaze-Informed and Context-Aware Pedestrian Trajectory Prediction in Shared Spaces with Automated Shuttles: A Virtual Reality Study
Department of Technology, Management and Economics at the Technical University of Denmark(丹麦技术大学技术、管理与经济学系);Department of Transport & Planning, Civil Engineering Geosciences at Delft University of Technology(代尔夫特理工大学交通运输与规划、土木工程与地质科学系)