Age-related cognitive impairment is generally characterized by gradual memory loss and decision-making difficulties. The aim of this study is to investigate multi level support and suggest relevant helping means for the elderly with mild cognitive impairment as well as their caregivers as the primary end-users. This work reports preliminary results on an intelligent assistive system, achieved through the integration of Internet of Things, augmented reality, and adaptive fuzzy decision-making methods. The proposed system operates in different modes, including automated and semi-automated modes. The former helps the user complete their daily life activities by showing augmented reality messages or making automatic changes; while the latter allows manual changes after the real-time assessment of the user's cognitive state based on the augmented reality serious game score. We have also evaluated the accuracy of the serious game score with 37 elderly participants and compared it with users' paper-based cognitive test results. We further noted that there is an acceptable correlation between the paper-based test and users' serious game scores. Moreover, we observed that the system response in the semi-automated mode causes less data loss compared with the automated mode, as the number of active devices decreases.
User-Centered Design of Socially Assistive Robotic Combined with Non-Immersive Virtual Reality-based Dyadic Activities for Older Adults Residing in Long Term Care Facilities
Functional upper-limb prosthetic training can improve users performance in controlling prostheses and has been incorporated into occupational therapy for individuals in need. In recent years, virtual reality (VR) and augmented reality (AR) technologies have been shown to be promising avenues to improve the convenience of rehabilitative prosthesis training systems. However, it is uncertain if the comprehensive efficacy and effectiveness of VR or AR assistive tools are adequate compared to conventional prosthetic tools and if not, whether enhancements can be made through incorporation of other t…
ChatMuse: Supporting In-Person Small-Group Conversation Experience with a Proactive Assistive AI Agent in Mixed Reality
Florida International University（佛罗里达国际大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Michigan（密歇根大学）；University of Arizona（亚利桑那大学）；University of Cincinnati（辛辛那提大学）
In this paper we describe and evaluate a mixed reality system that aims to augment users in task guidance applications by combining automated and unsupervised information collection with minimally invasive video guides. The result is a self-contained system that we call GlaciAR (Glass-enabled Contextual Interactions for Augmented Reality), that operates by extracting contextual interactions from observing users performing actions. GlaciAR is able to i) automatically determine moments of relevance based on a head motion attention model, ii) automatically produce video guidance information, iii)…
The objective of this paper is to present a Mixed Reality System (MRS) for rehabilitation of the upper limb after stroke. The system answers the following challenges: (i) increase motivation of patients by making the training a personalized experience; (ii) take into account patients' impairments by offering intuitive and easy to use interaction modalities; (iii) make it possible to therapists to track patient's activities and to evaluate/track their progress; (iv) open opportunities for telemedicine and tele rehabilitation; (v) and provide an economically acceptable system by reducing both eq…
To promote empathy with people that have disabilities, we propose a multi-sensory interactive experience that allows sighted users to embody having a visual impairment whilst using assistive technologies. The experiment involves blindfolded sighted participants interacting with a variety of sonification methods in order to locate targets and place objects in a real kitchen environment. Prior to the tests, we enquired about the perceived benefits of increasing said empathy from the blind and visually impaired (BVI) community. To test empathy, we adapted an Empathy and Sympathy Response scale to…
This is the preprint version of our paper on 3rd International Workshop on Virtual and Augmented Assistive Technology (VAAT) at IEEE Virtual Reality 2015 (VR2015). An assistive training tool for rehabilitation of dysphonic patients is designed and developed according to the practical clinical needs. The assistive tool employs a space flight game as the attractive logic part, and microphone arrays as input device, which is getting rid of ambient noise by setting a specific orientation. The therapist can guide the patient to play the game as well as the voice training simultaneously side by side…
Virtual Reality (VR) for Serious Games (SGs) is attracting increasing attention for training applications due to its potential to provide significantly enhanced learning to users. Some examples of the application of VR for SGs are complex training evacuation problems such as indoor earthquake evacuation or fire evacuation. The indoor 3D geometry of existing buildings can largely influence evacuees' behaviour, being instrumental in the design of VR SGs storylines and simulation scenarios. The VR scenarios of existing buildings can be generated from drawings and models. However, these data may n…
Head-mounted displays (HMDs) are popular immersive tools in general, not limited to entertainment but also for education, military, and serious games for health. While these displays have strong popularity, they still have user experience issues, triggering possible symptoms of discomfort to users. This condition is known as cybersickness (CS) and is one of the most popular research topics tied to virtual reality (VR) issues. We first present the main strategies focused on minimizing cybersickness problems in virtual reality. Following this, we propose a guideline framework based on CS causes …
This is the preprint version of our paper on 2015 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth2015). An assistive training tool software for rehabilitation of dysphonic patients is evaluated according to the practical clinical feedback from the treatments. One stroke sufferer and one parkinson sufferer have provided earnest suggestions for the improvement of our tool software. The assistive tool employs a serious game as the attractive logic part, and running on the tablet with normal microphone as input device. Seven pitch estimation algorit…
Future astronauts living and working on the Moon will face extreme environmental conditions impeding their operational safety and performance. While it has been suggested that Augmented Reality (AR) Head-Up Displays (HUDs) could potentially help mitigate some of these adversities, the applicability of AR in the unique lunar context remains underexplored. To address this limitation, we have produced an accurate representation of the lunar setting in virtual reality (VR) which then formed our testbed for the exploration of prospective operational scenarios with aerospace experts. Herein we prese…
Assistive technology is a prerequisite for making a high-quality lecture video. It is therefore imperative to edit the lecture video after recording. In this study, we aim to reduce the cumbersome task of lecture video editing by developing a system that enables the addition of visual effects in the video while recording. In particular, we use augmented reality (AR) technology to digitize and display in real-time lecture materials, assistant agents, and other recording contents used by the lecturer. Our system realizes such a mechanism as a lecture recording environment. In addition, our syste…
Phobia is a widespread mental illness, and severe phobias can seriously impact patients daily lives. One-session Exposure Treatment (OST) has been used to treat phobias in the early days,but it has many disadvantages. As a new way to treat a phobia, virtual reality exposure therapy(VRET) based on serious games is introduced. There have been much researches in the field of serious games for phobia therapy (SGPT), so this paper presents a detailed review of SGPT from three perspectives. First, SGPT in different stages has different forms with the update and iteration of technology. Therefore, we…
University of Waterloo（滑铁卢大学）；Vision and Image Processing Lab（视觉与图像处理实验室）
The ability to interpret social cues comes naturally for most people, but for those living with Autism Spectrum Disorder (ASD), some experience a deficiency in this area. This paper presents the development of a multimodal augmented reality (AR) system which combines the use of computer vision and deep convolutional neural networks (CNN) in order to assist individuals with the detection and interpretation of facial expressions in social settings. The proposed system, which we call AEGIS (Augmented-reality Expression Guided Interpretation System), is an assistive technology deployable on a vari…
In the recent years, various 3D mixed reality serious games have been developed for different applications such as physical training, rehabilitation, and education. Task performance in a serious game is a measurement of how efficiently and accurately users accomplish the game's objectives. Prior research includes a graph-based representation of tasks, e.g. Hierarchical Task Network (HTN), which only models a game's tasks but does not perform assessment. In this paper, we propose Assessment HTN (A-HTN), which both models the task efficiently and incorporates assessment logic for game objectives…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Colorado School of Mines（科罗拉多矿业学院）；IBM Research AI（IBM人工智能研究院）
Virtual, Augmented, and Mixed Reality for Human-Robot Interaction (VAM-HRI) has been gaining considerable attention in research in recent years. However, the HRI community lacks a set of shared terminology and framework for characterizing aspects of mixed reality interfaces, presenting serious problems for future research. Therefore, it is important to have a common set of terms and concepts that can be used to precisely describe and organize the diverse array of work being done within the field. In this paper, we present a novel taxonomic framework for different types of VAM-HRI interfaces, c…
Early adolescence is a time of major social change; a strong sense of belonging and peer connectedness is an essential protective factor in mental health during that period. In this paper we introduce LINA, an augmented reality (AR) smartphone-based serious game played in school by an entire class (age 10+) together with their teacher, which aims to facilitate and improve peer interaction, sense of belonging and class climate, while creating a safe space to reflect on mental health and external stressors related to family circumstance. LINA was developed through an interdisciplinary collaborat…
Development and Usability Study of Older Adults in Motion-Captured Serious Game Incorporating Olfactory Stimulations
Pathfinder Technology Limited（百胜科技有限公司）；Faculty of Science and Technology, UOW College Hong Kong（香港科技大学学院理科学院）；Department of Digital Innovation and Technology, Technological and Higher Education Institute of Hong Kong（香港高等教育科技学院数字创新与技术系）
Obesity is a serious disease that can affect both physical and psychological well-being. Due to weight stigmatization, many affected individuals suffer from body image disturbances whereby they perceive their body in a distorted way, evaluate it negatively, or neglect it. Beyond established interventions such as mirror exposure, recent advancements aim to complement body image treatments by the embodiment of visually altered virtual bodies in virtual reality (VR). We present a high-fidelity prototype of an advanced VR system that allows users to embody a rapidly generated personalized, photore…
Virtual reality (VR) technology is commonly used in entertainment applications; however, it has also been deployed in practical applications in more serious aspects of our lives, such as safety. To support people working in dangerous industries, VR can ensure operators manipulate standardized tasks and work collaboratively to deal with potential risks. Surprisingly, little research has focused on how people can collaboratively work in VR environments. Few studies have paid attention to the cognitive load of operators in their collaborative tasks. Once task demands become complex, many research…
The rapid development of technology has introduced new formats of human-computer interaction, which have in turn produced many new forms of media and a whole new field of interactive multimedia. One of the major mediums that has grown in popularity since its early development is video games. For a long time, video games have been developed and distributed for the purpose of entertainment, however, in the late 2010s, researchers have taken an interest in the characteristics of games and how they can be used for different purposes. Video games allow a tight loop of action-reaction which provides…
We aim to help users estimate the state of the world in tasks like robotic teleoperation and navigation with visual impairments, where users may have systematic biases that lead to suboptimal behavior: they might struggle to process observations from multiple sensors simultaneously, receive delayed observations, or overestimate distances to obstacles. While we cannot directly change the user's internal beliefs or their internal state estimation process, our insight is that we can still assist them by modifying the user's observations. Instead of showing the user their true observations, we syn…
