SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VR
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）
Zhimin Wang, Chenyu Gu, Feng Lu
备注：
Virtual reality, spatial interaction, intent recognition, agent-based execution, large language models
眼手协调交互正成为虚拟现实（VR）用户界面中的一种主流交互模式。当前这种多模态交互的范式要求用户学习预定义的手势并记忆多个手势-任务关联，可以总结为一个『操作到意图』的范式。这种范式增加了用户的学习成本，并且交互错误容忍度低。在本文中，我们提出SIAgent，一种新的『意图到操作』框架，允许用户通过基于常识和习惯的自然眼手运动来表达交互意图。我们的系统有两个主要组成部分：（1）意图识别，将空间交互数据转换为自然语言并推断用户意图；（2）基于代理的执行，生成一个代理来执行相应任务。这消除了记忆手势的需要，并且能够适应个人运动偏好，具有高错误容忍度。我们进行了两项用户研究，涵盖超过60个交互任务，将我们的方法与两种『操作到意图』技术进行比较。结果表明，我们的方法在意图识别准确率上优于注视+捏合交互（97.2% vs 93.1%），同时减少了手臂疲劳并提高了可用性，以及用户偏好。另一项研究验证了眼注视和手运动通道在意图识别中的功能。我们的工作为通过意图驱动设计提升VR交互智能提供了有价值的见解。我们的源代码和LLM提示将在发表时提供。
英文摘要：
Eye-hand coordinated interaction is becoming a mainstream interaction modality in Virtual Reality (VR) user interfaces.Current paradigms for this multimodal interaction require users to learn predefined gestures and memorize multiple gesture-task associations, which can be summarized as an ``Operation-to-Intent" paradigm. This paradigm increases users' learning costs and has low interaction error tolerance. In this paper, we propose SIAgent, a novel "Intent-to-Operation" framework allowing users to express interaction intents through natural eye-hand motions based on common sense and habits. Our system features two main components: (1) intent recognition that translates spatial interaction data into natural language and infers user intent, and (2) agent-based execution that generates an agent to execute corresponding tasks. This eliminates the need for gesture memorization and accommodates individual motion preferences with high error tolerance. We conduct two user studies across over 60 interaction tasks, comparing our method with two "Operation-to-Intent" techniques. Results show our method achieves higher intent recognition accuracy than gaze + pinch interaction (97.2% vs 93.1%) while reducing arm fatigue and improving usability, and user preference. Another study verifies the function of eye gaze and hand motion channels in intent recognition. Our work offers valuable insights into enhancing VR interaction intelligence through intent-driven design. Our source code and LLM prompts will be made available upon publication.
Tokyo Institute of Technology（东京工业大学）；AIST Japan（日本国立研究开发法人产业技术综合研究所）
An intent expression system using eye robots is proposed for a mascot robot system from a viewpoint of humatronics. The eye robot aims at providing a basic interface method for an information terminal robot system. To achieve better understanding of the displayed information, the importance and the degree of certainty of the information should be communicated along with the main content. The proposed intent expression system aims at conveying this additional information using the eye robot system. Eye motions are represented as the states in a pleasure-arousal space model. Changes in the model…
Huazhong University of Science and Technology（华中科技大学）；Nagoya University（名古屋大学）
Eye movement is closely related to limb actions, so it can be used to infer movement intentions. More importantly, in some cases, eye movement is the only way for paralyzed and impaired patients with severe movement disorders to communicate and interact with the environment. Despite this, eye-tracking technology still has very limited application scenarios as an intention recognition method. The goal of this paper is to achieve a natural fixation-based grasping intention recognition method, with which a user with hand movement disorders can intuitively express what tasks he/she wants to do by …
Conducting collaborative tasks, e.g., multi-user game, in virtual reality (VR) could enable us to explore more immersive and effective experience. However, for current VR systems, users cannot communicate properly with each other via their gaze points, and this would interfere with users' mutual understanding of the intention. In this study, we aimed to find the optimal eye tracking data visualization , which minimized the cognitive interference and improved the understanding of the visual attention and intention between users. We designed three different eye tracking data visualizations: gaze…
We have pioneered the Where-You-Look-Is Where-You-Go approach to controlling mobility platforms by decoding how the user looks at the environment to understand where they want to navigate their mobility device. However, many natural eye-movements are not relevant for action intention decoding, only some are, which places a challenge on decoding, the so-called Midas Touch Problem. Here, we present a new solution, consisting of 1. deep computer vision to understand what object a user is looking at in their field of view, with 2. an analysis of where on the object's bounding box the user is looki…
Anticipation Before Action: EEG-Based Implicit Intent Detection for Adaptive Gaze Interaction in Mixed Reality
LMU Munich（慕尼黑大学）；Carl von Ossietzky Universität Oldenburg（奥尔登堡卡尔·冯·奥西茨基大学）；TU Dortmund University（多特蒙德工业大学）；Research Center Trustworthy Data Science and Security（可信数据科学与安全研究中心）
The application of psychophysiologicy in human-computer interaction is a growing field with significant potential for future smart personalised systems. Working in this emerging field requires comprehension of an array of physiological signals and analysis techniques. Eye tracking is a widely used method for tracking user attention with gaze location, but also provides information on the internal cognitive and contextual state, intention, and the locus of the user's visual attention in interactive settings through a number of eye and eyelid movement related parameters. This paper presents a sh…
RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation
College of Engineering Northeastern University Boston, USA；Khoury College of Computer Sciences Northeastern University Boston, USA；Department of Civil \& Environmental Engineering Northeastern University Boston, USA
The P300 speller is being considered as an independent brain-computer interface. That means it measures the user's intent, and does not require the user to move any muscles. In particular it should not require eye fixation of the desired character. However, it has been shown that posterior electrodes provide significant discriminative information, which is likely related to visual processing. These findings imply the need for studies controlling the effect of eye movements. In experiments with a 3x3 character matrix, attention and eye fixation was directed to different characters. In the event…
Household robots need to communicate with human beings in a friendly fashion. To achieve better understanding of displayed information, an importance and a certainty of the information should be communicated together with the main information. The proposed intent expression system aims to convey this additional information using an eye robot. The eye motions are represented as states in a pleasure-arousal space model. Change of the model state is calculated by fuzzy inference according to the importance and certainty of the displayed information. This change influences the arousal-sleep coordi…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
What does human gaze reveal about a users' intents and to which extend can these intents be inferred or even visualized? Gaze was proposed as an implicit source of information to predict the target of visual search and, more recently, to predict the object class and attributes of the search target. In this work, we go one step further and investigate the feasibility of combining recent advances in encoding human gaze information using deep convolutional neural networks with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Such …
Indian Institute of Science（印度科学学院）；Indian Institute of Technology, Madras（马德拉斯印度理工学院）；Indian Institute of Information Technology, Kalyani（卡利亚尼印度信息技术学院）
Background: People with severe speech and motor impairment (SSMI) often uses a technique called eye pointing to communicate with outside world. One of their parents, caretakers or teachers hold a printed board in front of them and by analyzing their eye gaze manually, their intentions are interpreted. This technique is often error prone and time consuming and depends on a single caretaker. Objective: We aimed to automate the eye tracking process electronically by using commercially available tablet, computer or laptop and without requiring any dedicated hardware for eye gaze tracking. The eye …
MindEye-OmniAssist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution
Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology（脑启发智能系统湖北省重点实验室，华中科技大学）；Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（图像处理与智能控制教育部重点实验室，人工智能与自动化学院，华中科技大学）；state key laboratory of intelligent vehicle safety technology, chongqing changan automobile co ltd（智能车辆安全技术 state key laboratory，重庆长安汽车有限公司）；Science and technology innovation center, China ship development and design centre（科技创新中心，中国船舶工业设计研究中心）
The People's Gaze: Co-Designing and Refining Gaze Gestures with General Users and Gaze Interaction Experts
University of St Andrews（圣安德鲁斯大学）；University of Essex（埃塞克斯大学）；King’s College London（伦敦国王学院）；Dalian Maritime University（大连海事大学）；University of Glasgow（格拉斯哥大学）
Human eye gaze plays an important role in delivering information, communicating intent, and understanding others' mental states. Previous research shows that a robot's gaze can also affect humans' decision-making and strategy during an interaction. However, limited studies have trained humanoid robots on gaze-based data in human-robot interaction scenarios. Considering gaze impacts the naturalness of social exchanges and alters the decision process of an observer, it should be regarded as a crucial component in human-robot interaction. To investigate the impact of robot gaze on humans, we prop…
