Medical ultrasound has been widely used to examine vascular structure in modern clinical practice. However, traditional ultrasound examination often faces challenges related to inter- and intra-operator variation. The robotic ultrasound system (RUSS) appears as a potential solution for such challenges because of its superiority in stability and reproducibility. Given the complex anatomy of human vasculature, multiple vessels often appear in ultrasound images, or a single vessel bifurcates into branches, complicating the examination process. To tackle this challenge, this work presents a gaze-guided RUSS for vascular applications. A gaze tracker captures the eye movements of the operator. The extracted gaze signal guides the RUSS to follow the correct vessel when it bifurcates. Additionally, a gaze-guided segmentation network is proposed to enhance segmentation robustness by exploiting gaze information. However, gaze signals are often noisy, requiring interpretation to accurately discern the operator's true intentions. To this end, this study proposes a stabilization module to process raw gaze data. The inferred attention heatmap is utilized as a region proposal to aid segmentation and serve as a trigger signal when the operator needs to adjust the scanning target, such as when a bifurcation appears. To ensure appropriate contact between the probe and surface during scanning, an automatic ultrasound confidence-based orientation correction method is developed. In experiments, we demonstrated the efficiency of the proposed gaze-guided segmentation pipeline by comparing it with other methods. Besides, the performance of the proposed gaze-guided RUSS was also validated as a whole on a realistic arm phantom with an uneven surface.
MindEye-OmniAssist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution
Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology(脑启发智能系统湖北省重点实验室,华中科技大学);Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(图像处理与智能控制教育部重点实验室,人工智能与自动化学院,华中科技大学);state key laboratory of intelligent vehicle safety technology, chongqing changan automobile co ltd(智能车辆安全技术 state key laboratory,重庆长安汽车有限公司);Science and technology innovation center, China ship development and design centre(科技创新中心,中国船舶工业设计研究中心)
Action Anticipation: Reading the Intentions of Humans and Robots
Institute for Systems and Robotics(系统与机器人研究所);Instituto Superior Técnico(高等技术学院);Universidade de Lisboa(里斯本大学);Faculty of Technical Sciences(技术科学学院);University of Novi Sad(诺维萨德大学);University of Edinburgh(爱丁堡大学);School of Engineering(工程学院);EPFL(洛桑联邦理工学院)
The functional and temporal roles of gaze evolve across the phases and constraints of multi-stage robot-mediated manipulation
The BioRobotics Institute, Scuola Superiore Sant’Anna(生物机器人研究所,高等圣安娜学院);Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna(机器人与人工智能卓越部门,高等圣安娜学院)
RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation
College of Engineering Northeastern University Boston, USA;Khoury College of Computer Sciences Northeastern University Boston, USA;Department of Civil \& Environmental Engineering Northeastern University Boston, USA