Objective: Micro-navigation poses challenges for blind and visually impaired individuals. They often need to ask for sighted assistance. We explored the feasibility of utilizing ChatGPT as a virtual assistant to provide navigation directions. Methods: We created a test set of outdoor and indoor micro-navigation scenarios consisting of 113 scene images and their human-generated text descriptions. A total of 412 way-finding queries and their expected responses were compiled based on the scenarios. Not all queries are answerable based on the information available in the scene image. "I do not know"response was expected for unanswerable queries, which served as negative cases. High level orientation responses were expected, and step-by-step guidance was not required. ChatGPT 4o was evaluated based on sensitivity (SEN) and specificity (SPE) under different conditions. Results: The default ChatGPT 4o, with scene images as inputs, resulted in SEN and SPE values of 64.8% and 75.9%, respectively. Instruction on how to respond to unanswerable questions did not improve SEN substantially but SPE increased by around 14 percentage points. SEN and SPE both improved substantially, by about 17 and 16 percentage points on average respectively, when human written descriptions of the scenes were provided as input instead of images. Providing further prompt instructions to the assistants when the input was text description did not substantially change the SEN and SPE values. Conclusion: Current native ChatGPT 4o is still unable to provide correct micro-navigation guidance in some cases, probably because its scene understanding is not optimized for navigation purposes. If multi-modal chatbots could interpret scenes with a level of clarity comparable to humans, and also guided by appropriate prompts, they may have the potential to provide assistance to visually impaired for micro-navigation.
Outdoor shopping complexes (OSC) are extremely difficult for people with visual impairment to navigate. Existing GPS devices are mostly designed for roadside navigation and seldom transition well into an OSC-like setting. We report our study on the challenges faced by a blind person in navigating OSC through developing a new mobile application named iExplore. We first report an exploratory study aiming at deriving specific design principles for building this system by learning the unique challenges of the problem. Then we present a methodology that can be used to derive the necessary informati…
Today, many of the aid systems deployed for visually impaired people are mostly made for a single purpose. Be it navigation, object detection, or distance perceiving. Also, most of the deployed aid systems use indoor navigation which requires a pre-knowledge of the environment. These aid systems often fail to help visually impaired people in the unfamiliar scenario. In this paper, we propose an aid system developed using object detection and depth perceivement to navigate a person without dashing into an object. The prototype developed detects 90 different types of objects and compute their di…
National Taiwan University（国立台湾大学）；Ming Chuan University（铭传大学）
This work introduces Guardian Angel, an Android App that assists visually impaired people to avoid danger in complex traffic environment. The system, consisting of object detection by pretrained YOLO model, distance estimation and moving direction estimation, provides information about surrounding vehicles and alarms users of potential danger without expensive special purpose device. With an experiment of 8 subjects, we corroborate that in terms of satisfaction score in pedestrian-crossing experiment with the assistance of our App using a smartphone is better than when without under 99% confid…
Accessibility of websites for visually impaired persons is mishandled by screen readers which are not always adapted to interactivity needed by actual web/multimedia technologies. This paper analyses the difficulties to access to information with the use of different technologies and the existing recommendations for the conception of accessible websites for all. Preliminary results with visually impaired persons on our ACCESSPACE project website are presented.
The University of the South Pacific（南太平洋大学）；Software Foundation（软件基金会）
It is estimated that 285 million people globally are visually impaired. A majority of these people live in developing countries and are among the elderly population. One of the most difficult tasks faced by the visually impaired is identification of people. While naturally, voice recognition is a common method of identification, it is an intuitive and difficult process. The rise of computation capability of mobile devices gives motivation to develop applications that can assist visually impaired persons. With the availability of mobile devices, these people can be assisted by an additional met…
Beihang University（北京航空航天大学）；CloudMinds Technologies Inc.（达闼科技有限公司）；China Academy of Telecommunication Technology（中国电信技术研究院）
This paper presents a wearable assistive device with the shape of a pair of eyeglasses that allows visually impaired people to navigate safely and quickly in unfamiliar environment, as well as perceive the complicated environment to automatically make decisions on the direction to move. The device uses a consumer Red, Green, Blue and Depth (RGB-D) camera and an Inertial Measurement Unit (IMU) to detect obstacles. As the device leverages the ground height continuity among adjacent image frames, it is able to segment the ground from obstacles accurately and rapidly. Based on the detected ground,…
Access to textual and visual information for visually impaired persons becomes very difficult with screen readers which are not adapted to different websites.This paper analyses the use of different technologies for access digital content and to establish some ameliorations to the existing recommendations to accessible website conception for all. The preliminary evaluation results with visually impaired people of our website ACCESSPACE which is constructed with the existing recommendations, confirm the project's relevance.
This paper describes the ongoing development of a conversational interaction concept that allows visually impaired users to easily create and edit text documents on mobile devices using mainly voice input. In order to verify the concept, a prototype app was developed and tested for both iOS and Android systems, based on the natural-language understanding (NLU) platform Google Dialogflow. The app and interaction concept were repeatedly tested by users with and without visual impairments. Based on their feedback, the concept was continuously refined, adapted and improved on both mobile platforms…
In this paper a concept for hardware realization of graphic tactile display for visually impaired peoples is presented. For realization of tactile actuators bi-stable, solenoids and PIC based control board are used. The selected algorithm for series activation of each row of display allows using minimal number of active components to set and reset the solenoids. Finally, a program algorithm of control board is discussed. The project is funded by Bulgarian National Science Fund NSF Grant D ID 02 14, 2009 2013
Recent trend of touch-screen devices produces an accessibility barrier for visually impaired people. On the other hand, these devices come with sensors such as accelerometer. This calls for new approaches to human computer interface (HCI). In this study, our aim is to find an alternative approach to classify 20 different hand gestures captured by iPhone 3GS's built-in accelerometer and make high accuracy on user-independent classifications using Dynamic Time Warping (DTW) with dynamic warping window sizes. 20 gestures with 1,100 gesture data are collected from 15 normal-visioned people. This d…
Visually impaired people face important challenges related to orientation and mobility. Indeed, 56% of visually impaired people in France declared having problems concerning autonomous mobility. These problems often mean that visually impaired people travel less, which influences their personal and professional life and can lead to exclusion from society. Therefore this issue presents a social challenge as well as an important research area. Accessible geographic maps are helpful for acquiring knowledge about a city's or neighborhood's configuration, as well as selecting a route to reach a des…
Tactile maps are commonly used to give visually impaired users access to geographical representations. Although those relief maps are efficient tools for acquisition of spatial knowledge, they present several limitations and issues such as the need to read braille. Several research projects have been led during the past three decades in order to improve access to maps using interactive technologies. In this chapter, we present an exhaustive review of interactive map prototypes. We classified existing interactive maps into two categories: Digital Interactive Maps (DIMs) that are displayed on a …
Accessible pedestrian signal (APS) was proposed as a mean to achieve the same level of service that is set forth by the American with Disability Act (ADA) for the visually impaired. One of the major issues of existing APSs is the failure to deliver adequate crossing information for the visually impaired. This paper presents a mobile-based APS application, namely Virtual Guide Dog (VGD). Integrating intersection information and onboard sensors (e.g., GPS, compass, accelerometer, and gyroscope sensor) of modern smartphones, the VGD application can notify the visually impaired: 1) the close proxi…
Visually impaired people face numerous challenges when it comes to transportation. Not only must they circumvent obstacles while navigating, but they also need access to essential information related to available public transport, up-to-date weather forecast, and convenient method for booking private taxis. In this paper we introduce Transport Assistant - a voice based assistive technology prototype, built with a goal of leveling the playing field for the visually impaired to solve these problems that they face in their day to day life. Being voice enabled makes it seamlessly integrate into th…
The proposed system aims to be a techno-friend of visually impaired people to assist them in orientation and mobility both indoor and outdoor. Moving through an unknown environment becomes a real challenge for most of them, although they rely on their other senses. An age old mechanism used for assistance for the blind people is a white cane commonly known as walking cane a simple and purely mechanical device to detect the ground, uneven surfaces, holes and steps using simple Tactile-force feedback.
Currency is used almost everywhere to facilitate business. In most developing countries, especially the ones in Africa, tangible notes are predominantly used in everyday financial transactions. One of these countries, Ethiopia, is believed to have one of the world highest rates of blindness (1.6%) and low vision (3.7%). There are around 4 million visually impaired people; With 1.7 million people being in complete vision loss. Those people face a number of challenges when they are in a bus station, in shopping centers, or anywhere which requires the physical exchange of money. In this paper, we…
Multimodal interactive maps are a solution for presenting spatial information to visually impaired people. In this paper, we present an interactive multimodal map prototype that is based on a tactile paper map, a multi-touch screen and audio output. We first describe the different steps for designing an interactive map: drawing and printing the tactile paper map, choice of multi-touch technology, interaction technologies and the software architecture. Then we describe the method used to assess user satisfaction. We provide data showing that an interactive map - although based on a unique, elem…
Exploring an unfamiliar indoor environment and avoiding obstacles is challenging for visually impaired people. Currently, several approaches achieve the avoidance of static obstacles based on the mapping of indoor scenes. To solve the issue of distinguishing dynamic obstacles, we propose an assistive system with an RGB-D sensor to detect dynamic information of a scene. Once the system captures an image, panoptic segmentation is performed to obtain the prior dynamic object information. With sparse feature points extracted from images and the depth information, poses of the user can be estimated…
Recent studies have focused on developing advanced assistive devices to help blind or visually impaired people. Navigation is challenging for this community; however, developing a simple yet reliable navigation system is still an unmet need. This study targets the navigation problem and proposes a wearable assistive system. We developed a smart glove and shoe set based on radio-frequency identification technology to assist visually impaired people with navigation and orientation in indoor environments. The system enables the user to find the directions through audio feedback. To evaluate the d…
To overcome the travelling difficulty for the visually impaired group, this paper presents a novel ETA (Electronic Travel Aids)-smart guiding device in the shape of a pair of eyeglasses for giving these people guidance efficiently and safely. Different from existing works, a novel multi sensor fusion based obstacle avoiding algorithm is proposed, which utilizes both the depth sensor and ultrasonic sensor to solve the problems of detecting small obstacles, and transparent obstacles, e.g. the French door. For totally blind people, three kinds of auditory cues were developed to inform the directi…
Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；Université de Montréal（蒙特利尔大学）；Polytechnique Montréal（蒙特利尔高等理工学院）；McGill University（麦吉尔大学）
The number of visually impaired or blind (VIB) people in the world is estimated at several hundred million. Based on a series of interviews with the VIB and developers of assistive technology, this paper provides a survey of machine-learning based mobile applications and identifies the most relevant applications. We discuss the functionality of these apps, how they align with the needs and requirements of the VIB users, and how they can be improved with techniques such as federated learning and model compression. As a result of this study we identify promising future directions of research in …
TIB—Leibniz Information Centre for Science and Technology（莱布尼茨科学技术信息中心（TIB））；L3S Research Center（L3S研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）
Illustrations are widely used in education, and sometimes, alternatives are not available for visually impaired students. Therefore, those students would benefit greatly from an automatic illustration description system, but only if those descriptions were complete, correct, and easily understandable using a screenreader. In this paper, we report on a study for the assessment of automated image descriptions. We interviewed experts to establish evaluation criteria, which we then used to create an evaluation questionnaire for sighted non-expert raters, and description templates. We used this que…
Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functi…
With the large number of partially or completely visually impaired persons in society, their integration as productive, educated and capable members of society is hampered heavily by a pervasively high level of braille illiteracy. This problem is further compounded by the fact that braille printers are prohibitively expensive - generally starting from two thousand US dollars, beyond the reach of the common man. Over the period of a year, the authors have tried to develop a Braille printer which attempts to overcome the problems inherent in commercial printers. The purpose of this paper, theref…
To promote empathy with people that have disabilities, we propose a multi-sensory interactive experience that allows sighted users to embody having a visual impairment whilst using assistive technologies. The experiment involves blindfolded sighted participants interacting with a variety of sonification methods in order to locate targets and place objects in a real kitchen environment. Prior to the tests, we enquired about the perceived benefits of increasing said empathy from the blind and visually impaired (BVI) community. To test empathy, we adapted an Empathy and Sympathy Response scale to…
Lacking the ability to sense ambient environments effectively, blind and visually impaired people (BVIP) face difficulty in walking outdoors, especially in urban areas. Therefore, tools for assisting BVIP are of great importance. In this paper, we propose a novel "flying guide dog" prototype for BVIP assistance using drone and street view semantic segmentation. Based on the walkable areas extracted from the segmentation prediction, the drone can adjust its movement automatically and thus lead the user to walk along the walkable path. By recognizing the color of pedestrian traffic lights, our p…
A sudden reliance on the internet has resulted in the global standardization of specific software and interfaces tailored for the average user. Whether it be web apps or dedicated software, the methods of interaction are seemingly similar. But when the computer tool is presented with unique users, specifically with a disability, the quality of interaction degrades, sometimes to a point of complete uselessness. This roots from one's focus on the average user rather than the development of a platform for all (a golden standard). This paper reviews published works and products that deal with prov…
Sighted players gain spatial awareness within video games through sight and spatial awareness tools (SATs) such as minimaps. Visually impaired players (VIPs), however, must often rely heavily on SATs to gain spatial awareness, especially in complex environments where using rich ambient sound design alone may be insufficient. Researchers have developed many SATs for facilitating spatial awareness within VIPs. Yet this abundance disguises a gap in our understanding about how exactly these approaches assist VIPs in gaining spatial awareness and what their relative merits and limitations are. To a…
VIT-AP University（VIT-AP大学）；University of North Texas（北德克萨斯大学）
Individuals with visual impairments often face a multitude of challenging obstacles in their daily lives. Vision impairment can severely impair a person's ability to work, navigate, and retain independence. This can result in educational limits, a higher risk of accidents, and a plethora of other issues. To address these challenges, we present MagicEye, a state-of-the-art intelligent wearable device designed to assist visually impaired individuals. MagicEye employs a custom-trained CNN-based object detection model, capable of recognizing a wide range of indoor and outdoor objects frequently en…
According to the World Health Organization, visual impairment is estimated to affect approximately 2.2 billion people worldwide. The visually impaired must currently rely on navigational aids to replace their sense of sight, like a white cane or GPS (Global Positioning System) based navigation, both of which fail to work well indoors. The white cane cannot be used to determine a user's position within a room, while GPS can often lose connection indoors and does not provide orientation information, making both approaches unsuitable for indoor use. Therefore, this research seeks to develop a 3D-…
The number of applications on online mobile application stores is increasing at a rapid rate. Smart-phones are used by a wide range of people varying in age, and also in the ability to use a smart phone. With the increasing dependency on smart-phones, the paper aims to determine whether the popular applications on Google Play, the official store for Android applications, can be used by people with vision impairment. The accessibility of the applications was tested using an external keyboard, and TalkBack, an accessibility tool developed by Google. It was found that several popular applications…
People with special needs like blind and visually impaired (BVI) people can particularly benefit from using voice assistants providing spoken information input and output in everyday life. However, it is crucial to understand their needs and to include these in the development of accessible and useful assistance systems. By conducting an online survey with 145 BVI people, this paper revealed that common voice assistants like Apple's Siri or Amazon's Alexa are used by a majority of BVI people and are also considered helpful. In particular, features in the context of audio entertainment, interne…
An estimated 253 million people have visual impairments. These visual impairments affect everyday lives, and limit their understanding of the outside world. This can pose a risk to health from falling or collisions. We propose a solution to this through quick and detailed communication of environmental spatial geometry through sound, providing the blind and visually impaired the ability to understand their spatial environment through sound technology. The model consists of fast object detection and 3D environmental mapping, which is communicated through a series of quick sound notes. These sou…
This work designs, evaluates, and improves a proposed search engine interface for Visually Impaired VI users to efficiently perform web search activities. Our conceptual modeling technique is based on Formal Concept Analysis FCA that is used for data analysis. This approach highlights the hierarchized approach to represent the discovered concepts. It is combined with context interactive navigation in an interface which is called interactive search engine (InteractSE). This interface aims to reduce the time and effort required by the VI users to browse search results. InteractSE was evaluated b…
