In this paper, we propose a deep learning based assistive system to improve the environment perception experience of visually impaired (VI). The system is composed of a wearable terminal equipped with an RGBD camera and an earphone, a powerful processor mainly for deep learning inferences and a smart phone for touch-based interaction. A data-driven learning approach is proposed to predict safe and reliable walkable instructions using RGBD data and the established semantic map. This map is also used to help VI understand their 3D surrounding objects and layout through well-designed touchscreen interactions. The quantitative and qualitative experimental results show that our learning based obstacle avoidance approach achieves excellent results in both indoor and outdoor datasets with low-lying obstacles. Meanwhile, user studies have also been carried out in various scenarios and showed the improvement of VI's environment perception experience with our system.
Assistive devices are a key aspect in wearable systems for biomedical applications, as they represent potential aids for people with physical and sensory disabilities that might lead to improvements in the quality of life. This chapter focuses on wearable assistive devices for the blind. It intends to review the most significant work done in this area, to present the latest approaches for assisting this population and to understand universal design concepts for the development of wearable assistive devices and systems for the blind.
The coupling of human movement dynamics with the function and design of wearable assistive devices is vital to better understand the interaction between the two. Advanced neuromuscular models and optimal control formulations provide the possibility to study and improve this interaction. In addition, optimal control can also be used to generate predictive simulations that generate novel movements for the human model under varying optimization criterion.
Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging
Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；School of Information and Electronic, Beijing Institute of Technology（北京理工大学信息与电子学院）；School of Mechanical Engineering, Tongji University（同济大学机械工程学院）
本文提出部署于Jetson Orin NX的视觉可穿戴转向辅助系统，通过轻量多任务网络检测跑道线与障碍物，并结合采样和样条曲线规划，帮助视障者安全慢跑。
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人形机器人与智能系统研究所）；Hangzhou SurImage Technology Company Ltd.（杭州 SurImage 科技有限公司）；Hangzhou KrVision Technology Company Ltd.（杭州 KrVision 科技有限公司）
As the scene information, including objectness and scene type, are important for people with visual impairment, in this work we present a multi-task efficient perception system for the scene parsing and recognition tasks. Building on the compact ResNet backbone, our designed network architecture has two paths with shared parameters. In the structure, the semantic segmentation path integrates fast attention, with the aim of harvesting long-range contextual information in an efficient manner. Simultaneously, the scene recognition path attains the scene type inference by passing the semantic feat…
When we develop wearable assistive devices comfort and support are two main issues needed to be considered. In conventional design approaches, the degree of freedom of wearer's joint movement tends to be oversimplified. Accordingly, the wearer's motion becomes restrained and bone/ligament injuries might occur in case of unexpected fall. To mitigate those issues, this letter proposes a novel joint link mechanism inspired by a human spine structure as well as functionalities. The key feature of the proposed spine-like joint link mechanism is that hemispherical blocks are concatenated via flexibl…
Robotic sensorimotor extensions (supernumerary limbs, prosthesis, handheld tools) are worn devices used to interact with the nearby environment, whether to assist the capabilities of impaired users or to enhance the dexterity of industrial operators. Despite numerous mechanical achievements, embedding these robotics devices remains critical due to their weight and discomfort. To emancipate from these mechanical constraints, we propose a new hybrid system using a virtually worn robotic arm in augmented-reality, and a real robotic manipulator servoed on such virtual representation. We aim at bri…
Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functi…
Mobile manipulators in the home can provide increased autonomy to individuals with severe motor impairments, who often cannot complete activities of daily living (ADLs) without the help of a caregiver. Teleoperation of an assistive mobile manipulator could enable an individual with motor impairments to independently perform self-care and household tasks, yet limited motor function can impede one's ability to interface with a robot. In this work, we present a unique inertial-based wearable assistive interface, embedded in a familiar head-worn garment, for individuals with severe motor impairmen…
The physically impaired commonly have difficulties performing simple routine tasks without relying on other individuals who are not always readily available and thus make them strive for independence. While their impaired abilities can in many cases be augmented (to certain degrees) with the use of assistive technologies, there has been little attention to their applications in embodied AI with assistive technologies. This paper presents the modular framework, architecture, and design of the mid-fidelity prototype of MARVIN: an artificial-intelligence-powered robotic assistant designed to help…
Sight Guide: A Wearable Assistive Perception and Navigation System for the Vision Assistance Race in the Cybathlon 2024
Autonomous Systems Lab, ETH Zürich（苏黎世联邦理工学院自主系统实验室）；Robotics and Perception Group, University of Zurich（苏黎世大学机器人与感知组）；School of Engineering, Zurich University of Applied Sciences（应用科学大学工程学院）；Robotics Systems Lab, ETH Zürich（苏黎世联邦理工学院机器人系统实验室）
Common fully glazed facades and transparent objects present architectural barriers and impede the mobility of people with low vision or blindness, for instance, a path detected behind a glass door is inaccessible unless it is correctly perceived and reacted. However, segmenting these safety-critical objects is rarely covered by conventional assistive technologies. To tackle this issue, we construct a wearable system with a novel dual-head Transformer for Transparency (Trans4Trans) model, which is capable of segmenting general and transparent objects and performing real-time wayfinding to assis…
Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated…
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人机与机器人研究所）；Sino-German School, Tongji University（同济大学中德学院）；Hangzhou SurImage Technology Company Ltd.（杭州 SurImage 科技有限公司）；Hangzhou KrVision Technology Company Ltd.（杭州 KrVision 科技有限公司）
Classic computer vision algorithms, instance segmentation, and semantic segmentation can not provide a holistic understanding of the surroundings for the visually impaired. In this paper, we utilize panoptic segmentation to assist the navigation of visually impaired people by offering both things and stuff awareness in the proximity of the visually impaired efficiently. To this end, we propose an efficient Attention module -- Lintention which can model long-range interactions in linear time using linear space. Based on Lintention, we then devise a novel panoptic segmentation model which we ter…
Intelligent assistive systems can navigate blind people, but most of them could only give non-intuitive cues or inefficient guidance. Based on computer vision and vibrotactile encoding, this paper presents an interactive system that provides blind people with intuitive spatial cognition. Different from the traditional auditory feedback strategy based on speech cues, this paper firstly introduces a vibration-encoded feedback method that leverages the haptic neural pathway and enables the users to interact with objects other than manipulating an assistance device. Based on this strategy, a weara…
Consider an assistive system that guides visually impaired users through speech and haptic feedback to their destination. Existing robotic and ubiquitous navigation technologies (e.g., portable, ground, or wearable systems) often operate in a generic, user-agnostic manner. However, to minimize confusion and navigation errors, our real-world analysis reveals a crucial need to adapt the instructional guidance across different end-users with diverse mobility skills. To address this practical issue in scalable system design, we propose a novel model-based reinforcement learning framework for perso…
OpenGuide: Assistive Object Retrieval in Indoor Spaces for Individuals with Visual Impairments
Department of Civil and Environmental Engineering, University of Michigan（密歇根大学土木与环境工程系）；College of Literature, Science, and the Arts, University of Michigan（密歇根大学文学、科学与艺术学院）
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Guiding robots, in the form of canes or cars, have recently been explored to assist blind and low vision (BLV) people. Such robots can provide full or partial autonomy when guiding. However, the pros and cons of different forms and autonomy for guiding robots remain unknown. We sought to fill this gap. We designed autonomy-switchable guiding robotic cane and car. We conducted a controlled lab-study (N=12) and a field study (N=9) on BLV. Results showed that full autonomy received better walking performance and subjective ratings in the controlled study, whereas participants used more partial au…
Momentum-constrained Hybrid Heuristic Trajectory Optimization Framework with Residual-enhanced DRL for Visually Impaired Scenarios
School of Information；Software Engineering, University of Electronic Science；Sichuan Provincial Key Laboratory for Human Disease Gene Study, Sichuan Academy of Medical Sciences \& Sichuan Provincial People's Hospital, University of Electronic Science；Research Unit for Blindness Prevention, Chinese Academy of Medical Sciences, Sichuan Academy of Medical Sciences；Sichuan Provincial People’s Hospital, Chengdu, Sichuan, China.
Imperial College London（帝国理工学院）；MRC London Institute of Medical Sciences（英国医学研究理事会伦敦医学科学研究所）；Data Science Institute（数据科学研究所）
Assistive and Wearable Robotics have the potential to support humans with different types of motor impairments to become independent and fulfil their activities of daily living successfully. The success of these robot systems, however, relies on the ability to meaningfully decode human action intentions and carry them out appropriately. Neural interfaces have been explored for use in such system with several successes, however, they tend to be invasive and require training periods in the order of months. We present a robotic system for human augmentation, capable of actuating the user's arm an…
RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation
College of Engineering Northeastern University Boston, USA；Khoury College of Computer Sciences Northeastern University Boston, USA；Department of Civil \& Environmental Engineering Northeastern University Boston, USA
