四肢瘫痪和类似瘫痪形式的个体由于缺乏自主性而在身心上遭受痛苦。为了帮助恢复部分自主性,辅助机械臂已被证明能够提高生活独立性。然而,瘫痪用户对这些设备的控制提出了独特的挑战条件。在本文中,我们提出了叙事性图形用户界面的使用,这是一种应用于机器人的眼动控制界面的新颖、直观且计算成本低廉的方法。通过使用与基准标记配对的符号,可以在现实世界中定义用户可通过眼动触发的交互式按钮,并且这些按钮可以轻松嵌入到环境中。我们应用该系统来驾驶一个3自由度机械臂进行精确的拾取与放置任务。界面直接放置在机器人上,以实现直观和直接的交互,消除了在外部屏幕、菜单和机器人之间进行上下文切换的需要。经过校准和短暂的适应期,来自不同背景、年龄和视力条件的21名参与者完成了Yale-CMU-Berkeley (YCB) Block Pick and Place Protocol以对系统进行基准测试,在最高16.00分中取得了13.71分的平均分。报告显示系统具有良好的可用性和用户体验(System Usability Score为75.36),同时实现了较低的任务工作负荷测量值(NASA-TLX为44.76)。结果表明,用户可以通过最少的练习和较小的认知负荷来使用多个界面元素执行操作。据我们所知,这是第一个易于重新配置的无屏幕系统,能够完全通过眼动实现机器人控制以进行笛卡尔机器人控制,而无需眼部或面部手势。
英文摘要:
Individuals with tetraplegia and similar forms of paralysis suffer physically and emotionally due to a lack of autonomy. To help regain part of this autonomy, assistive robotic arms have been shown to increase living independence. However, users with paralysis pose unique challenging conditions for the control of these devices. In this article, we present the use of Diegetic Graphical User Interfaces, a novel, intuitive, and computationally inexpensive approach for gaze-controlled interfaces applied to robots. By using symbols paired with fiducial markers, interactive buttons can be defined in the real world which the user can trigger via gaze, and which can be embedded easily into the environment. We apply this system to pilot a 3-degree-of-freedom robotic arm for precision pick-and-place tasks. The interface is placed directly on the robot to allow intuitive and direct interaction, eliminating the need for context-switching between external screens, menus, and the robot. After calibration and a brief habituation period, twenty-one participants from multiple backgrounds, ages and eye-sight conditions completed the Yale-CMU-Berkeley (YCB) Block Pick and Place Protocol to benchmark the system, achieving a mean score of 13.71 out of the maximum 16.00 points. Good usability and user experience were reported (System Usability Score of 75.36) while achieving a low task workload measure (NASA-TLX of 44.76). Results show that users can employ multiple interface elements to perform actions with minimal practice and with a small cognitive load. To our knowledge, this is the first easily reconfigurable screenless system that enables robot control entirely via gaze for Cartesian robot control without the need for eye or face gestures.
MindEye-OmniAssist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution
Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology(脑启发智能系统湖北省重点实验室,华中科技大学);Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(图像处理与智能控制教育部重点实验室,人工智能与自动化学院,华中科技大学);state key laboratory of intelligent vehicle safety technology, chongqing changan automobile co ltd(智能车辆安全技术 state key laboratory,重庆长安汽车有限公司);Science and technology innovation center, China ship development and design centre(科技创新中心,中国船舶工业设计研究中心)
MIHRaGe: A Mixed-Reality Interface for Human-Robot Interaction via Gaze-Oriented Control
Engineering School of São Carlos University of São Paulo(圣保罗大学圣卡洛斯工程学院);Massachussets Institute of Technology(麻省理工学院);Hospital Israelita Albert Einstein(以色列医院阿尔伯特·爱因斯坦医院)
Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction
Technical University of Munich(慕尼黑技术大学);Robert Bosch GmbH, Corporate Research(博世集团,企业研究);Centre for Applied Autonomous Sensor Systems (AASS)(应用自主传感器系统中心(AASS))
A Multi-Layer Sim-to-Real Framework for Gaze-Driven Assistive Neck Exoskeletons
Kahlert School of Computing, University of Utah(犹他大学计算机学院);Dept. of Mechanical Engineering, University of Utah(犹他大学机械工程系);Robotics Center, University of Utah(犹他大学机器人中心)
Bidirectional Intent Communication: A Role for Large Foundation Models
TU Chemnitz(开姆尼茨工业大学);Robert Bosch GmbH(罗伯特·博世有限公司);Technical University of Munich(慕尼黑工业大学);Centre for Applied Autonomous Sensor Systems (AASS), Örebro University(厄勒布鲁大学应用自主传感系统中心)
RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation
College of Engineering Northeastern University Boston, USA;Khoury College of Computer Sciences Northeastern University Boston, USA;Department of Civil \& Environmental Engineering Northeastern University Boston, USA