This paper proposed a new interaction paradigm in the virtual reality (VR) environments, which consists of a virtual mirror or window projected onto a virtual surface, representing the correct perspective geometry of a mirror or window reflecting the real world. This technique can be applied to various videos, live streaming apps, augmented and virtual reality settings to provide an interactive and immersive user experience. To support such a perspective-accurate representation, we implemented computer vision algorithms for feature detection and correspondence matching. To constrain the solutions, we incorporated an automatically tuning scaling factor upon the homography transform matrix such that each image frame follows a smooth transition with the user in sight. The system is a real-time rendering framework where users can engage their real-life presence with the virtual space.
X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user's current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are also dynamically rendered based on the user's eye gaze. We build this X-ray vision window for a current state-of-the-art MR Head-Mounted Device (HMD) -- HoloLens 2 by integrating several different fea…
International Center for AI and Robot Autonomy（国际人工智能与机器人自主中心）
We present a congestion-aware routing solution for indoor evacuation, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees' locations. A population density map, obtained on-the-fly by aggregating locations of evacuees from user-end Augmented Reality (AR) devices, is used to model the congestion distribution inside a building. To efficiently search the evacuation route among all destinations, a variant of A* algorithm is devised to obtain the optimal solution in a single pass. In a series of simulated studies, we show …
Software analytics in augmented reality (AR) is said to have great potential. One reason why this potential is not yet fully exploited may be usability problems of the AR user interfaces. We present an iterative and qualitative usability evaluation with 15 subjects of a state-of-the-art application for software analytics in AR. We could identify and resolve numerous usability issues. Most of them were caused by applying conventional user interface elements, such as dialog windows, buttons, and scrollbars. The used city visualization, however, did not cause any usability issues. Therefore, we a…
Near distances are overestimated in virtual reality, and far distances are underestimated, but an explanation for these distortions remains elusive. One potential concern is that whilst the eye rotates to look at the virtual scene, the virtual cameras remain static. Could using eye-tracking to change the perspective of the virtual cameras as the eye rotates improve depth perception in virtual reality? This paper identifies 14 distinct perspective distortions that could in theory occur from keeping the virtual cameras fixed whilst the eye rotates in the context of near-eye displays. However, th…
University of Waterloo（滑铁卢大学）；Vision and Image Processing Lab（视觉与图像处理实验室）
The ability to interpret social cues comes naturally for most people, but for those living with Autism Spectrum Disorder (ASD), some experience a deficiency in this area. This paper presents the development of a multimodal augmented reality (AR) system which combines the use of computer vision and deep convolutional neural networks (CNN) in order to assist individuals with the detection and interpretation of facial expressions in social settings. The proposed system, which we call AEGIS (Augmented-reality Expression Guided Interpretation System), is an assistive technology deployable on a vari…
Augmented reality enables the illusion of contents such as objects and humans in the virtual world co-existing with users in the real world. However, non-users who are not aware of the presence of the virtual world and dynamically move nearby might either cause a conflict by directly breaking into space where a user is talking to a Virtual Human (VH), or be troubled when try to avoid disturbing the user. To maintain harmony and keep both the user's and non-users' comfort, we propose a method that controls the VH to adjust its own position to avoid such potential conflict. The difficulty to add…
Augmented Reality is the fusion of virtual components and our real surroundings. The simultaneous visibility of generated and natural objects often requires users to direct their selective attention to a specific target that is either real or virtual. In this study, we investigated whether this target is real or virtual by using machine learning techniques to classify electroencephalographic (EEG) data collected in Augmented Reality scenarios. A shallow convolutional neural net classified 3 second data windows from 20 participants in a person-dependent manner with an average accuracy above 70\…
Augmented reality provides new possibilities to propose environments where the designers can take advantage of the physicality of the artifacts while keeping the versatility of digital environments. Mixed objects can therefore provide new media in the interactions between stakeholders. Besides, the increasing interest in user participation in early design phases is limited by the poor representations or the expensive mock ups to be provided in design meetings. Therefore, understanding the role of these mixed artifacts by analyzing and characterizing the interactions is crucial to the developme…
Summary: Advances in 3D live cell microscopy are enabling high-resolution capture of previously unobserved processes. Unleashing the power of modern machine learning methods to fully benefit from these technologies is, however, frustrated by the difficulty of manually annotating 3D training data. MiCellAnnGELo virtual reality software offers an immersive environment for viewing and interacting with 4D microscopy data, including efficient tools for annotation. We present tools for labelling cell surfaces with a wide range of applications, including cell motility, endocytosis, and transmembrane …
This paper proposes the concept of "live-action virtual reality games" as a new genre of digital games based on an innovative combination of live-action, mixed-reality, context-awareness, and interaction paradigms that comprise tangible objects, context-aware input devices, and embedded/embodied interactions. Live-action virtual reality games are "live-action games" because a player physically acts out (using his/her real body and senses) his/her "avatar" (his/her virtual representation) in the game stage, which is the mixed-reality environment where the game happens. The game stage is a kind …
This paper addresses the problem of IDE interface complexity by introducing single-window graphical user interface. This approach lies in removing additional child windows from IDE, thus allowing a user to keep only text editor window open. We describe an abstract model of IDE GUI that is based on most popular modern integrated environments and has generalized user interface parts. Then this abstract model is reorganized into single windowed interface model: access to common IDE functions is provided from the code editing window while utility windows are removed without loss of IDE functionali…
This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop tim…
HandPad: A Bimanual Hand Interface for Fluid Window Interactions in VR
Department of Computer Science, University of Virginia（弗吉尼亚大学计算机科学系）；Department of Computer Science and Engineering, Ulsan National Institute of Science and Technology（蔚山科学技术院计算机科学与工程系）
Providing a depth-rich Virtual Reality (VR) experience to users without causing discomfort remains to be a challenge with today's commercially available head-mounted displays (HMDs), which enforce strict measures on stereoscopic camera parameters for the sake of keeping visual discomfort to a minimum. However, these measures often lead to an unimpressive VR experience with shallow depth feeling. We propose the first method ready to be used with existing consumer HMDs for automated stereoscopic camera control in virtual environments (VEs). Using radial basis function interpolation and projectio…
The objective of this study is to develop and use a virtual reality game as a tool to assess the effects of realistic stress on the behavioral and physiological responses of participants. The game is based on a popular Steam game called Keep Talking Nobody Explodes, where the player collaborates with another person to defuse a bomb. Varying levels of difficulties in solving a puzzle and time pressures will result in different stress levels that can be measured in terms of errors, response time lengths, and other physiological measurements. The game was developed using 3D programming tools incl…
Object locating in virtual reality (VR) has been widely used in many VR applications, such as virtual assembly, virtual repair, virtual remote coaching. However, when there are a large number of objects in the virtual environment(VE), the user cannot locate the target object efficiently and comfortably. In this paper, we propose a label guidance based object locating method for locating the target object efficiently in VR. Firstly, we introduce the label guidance based object locating pipeline to improve the efficiency of the object locating. It arranges the labels of all objects on the same s…
The virtual reality (VR) and human-computer interaction (HCI) combination has radically changed the way users approach a virtual environment, increasing the feeling of VR immersion, and improving the user experience and usability. The evolution of these two technologies led to the focus on VR locomotion and interaction. Locomotion is generally controller-based, but today hand gesture recognition methods were also used for this purpose. However, hand gestures can be stressful for the user who has to keep the gesture activation for a long time to ensure locomotion, especially continuously. Likew…
Physical keyboards are common peripherals for personal computers and are efficient standard text entry devices. Recent research has investigated how physical keyboards can be used in immersive head-mounted display-based Virtual Reality (VR). So far, the physical layout of keyboards has typically been transplanted into VR for replicating typing experiences in a standard desktop environment. In this paper, we explore how to fully leverage the immersiveness of VR to change the input and output characteristics of physical keyboard interaction within a VR environment. This allows individual physica…
ECG is an attractive option to assess stress in serious Virtual Reality (VR) applications due to its non-invasive nature. However, the existing Machine Learning (ML) models perform poorly. Moreover, existing studies only perform a binary stress assessment, while to develop a more engaging biofeedback-based application, multi-level assessment is necessary. Existing studies annotate and classify a single experience (e.g. watching a VR video) to a single stress level, which again prevents design of dynamic experiences where real-time in-game stress assessment can be utilized. In this paper, we re…
Virtual Reality is used successfully to treat people for regular phobias. A new challenge is to develop Virtual Reality Exposure Training for social skills. Virtual actors in such systems have to show appropriate social behavior including emotions, gaze, and keeping distance. The behavior must be realistic and real-time. Current approaches consist of four steps: 1) trainee social signal detection, 2) cognitive-affective interpretation, 3) determination of the appropriate bodily responses, and 4) actuation. The "cognitive" detour of such approaches does not match the directness of human bodily …
Fueled by advances in multi-party communications, increasingly mature immersive technologies being adopted, and the COVID-19 pandemic, a new wave of social virtual reality (VR) platforms have emerged to support socialization, interaction, and collaboration among multiple remote users who are integrated into shared virtual environments. Social VR aims to increase levels of (co-)presence and interaction quality by overcoming the limitations of 2D windowed representations in traditional multi-party video conferencing tools, although most existing solutions rely on 3D avatars to represent users. T…
Screen-based Augmented Reality (AR) systems can be built as a window into the real world as often done in mobile AR applications or using the Magic Mirror metaphor, where users can see themselves with augmented graphics on a large display. Such Magic Mirror systems have been used in digital clothing environments to create virtual dressing rooms, to teach human anatomy, and for collaborative design tasks. The term Magic Mirror implies that the display shows the users enantiomorph, i.e. the mirror image, such that the system mimics a real-world physical mirror. However, the question arises wheth…
This paper considers the problem of enabling the user to modify the path of a telepresence robot. The robot is capable of autonomously navigating to a goal predefined by the user, but the user might still want to modify the path, for example, to go further away from other people, or to go closer to landmarks she wants to see on the way. We propose Human-Influenced Dynamic Window Approach (HI-DWA), a shared control method aimed for telepresence robots based on Dynamic Window Approach (DWA) that allows the user to influence the control input given to the robot. To verify the proposed method, we …
A Virtual Reality Training System for Automotive Engines Assembly and Disassembly
Southern University of Science and Technology（南方科技大学）；Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（南方科技大学可信自主系统研究院）
