Research in user interaction with mixed reality environments using multiple displays has become increasingly relevant with the prevalence of mobile devices in everyday life and increased commoditization of large display area technologies using projectors or large displays. Previous work often combines touch-based input with other approaches, such as gesture-based input, to expand the possible interaction space or deal with limitations of other two-dimensional input methods. In contrast to previous methods, we examine the possibilities when the control-display (C-D) ratio is significantly smaller than one and small input movements result in large output movements. To this end one specific multi-display configuration is implemented in the form of a spatial-augmented reality sandbox environment, and used to explore various interaction techniques based on a variety of mobile device touch-based input and optical marker tracking-based finger input. A small pilot study determines the most promising input candidate, which is compared to traditional touch-input based techniques in a user study that tests it for practical relevance. Results and conclusions of the study are presented.
Immersive Analytics (IA) and consumer adoption of augmented reality (AR) and virtual reality (VR) head-mounted displays (HMDs) are both rapidly growing. When used in conjunction, stereoscopic IA environments can offer improved user understanding and engagement; however, it is unclear how the choice of stereoscopic display impacts user interactions within an IA environment. This paper presents a pilot study that examines the impact of stereoscopic display type on object manipulation and environmental navigation using consumer-available AR and VR displays. This work finds that the display type c…
Software libraries for most of the modern programming languages are numerous, large and complex. Remembering the syntax and usage of APIs is a difficult task for not just novices but also expert programmers. IDEs (Integrated Development Environment) provide capabilities like autocomplete and intellisense to assist programmers; however, programmers still need to visit search engines like Google to find API (Application Program Interface) documentation and samples. This paper evaluates Redprint - a browser based development environment for PHP that integrates API specific "Instant Example" and "…
Windows are used quite frequently in a GUI environment. The greatest advantage of using windows is that each window creates a virtual screen space. Hence, although the physical screen space is limited to a few inches, use of windows can create unlimited screen space to display innumerable items. The use of windows facilitates the user to open and interact with multiple programs or documents simultaneously in different windows. Sometimes a single program may also open multiple windows to display various items. The user can resize the windows and move their location time to time as desired. Howe…
Coordinated multi-display environments from the desktop, second-screen to gigapixel display walls are increasingly common. Personal and intimate mobile and wearable devices such as head-mounted displays, smartwatches, smartphones and tablets are rarely part of such multi-device ecosystems. With this paper, we contribute to a better understanding about factors that impede the creation and use of such mobile multi-device ecosystems. We base our findings on literature research and an expert survey. Specifically, we present grounded challenges relevant for the design, development and use of mobile…
We present our current research on the implementation of gaze as an efficient and usable pointing modality supplementary to speech, for interacting with augmented objects in our daily environment or large displays, especially immersive virtual reality environments, such as reality centres and caves. We are also addressing issues relating to the use of gaze as the main interaction input modality. We have designed and developed two operational user interfaces: one for providing motor-disabled users with easy gaze-based access to map applications and graphical software; the other for iteratively …
Modern display environments offer great potential for involving multiple users in presentations, discussions, and data analysis sessions. By showing multiple views on multiple displays, information exchange can be improved, several perspectives on the data can be combined, and different analysis strategies can be pursued. In this report, we describe concepts to support display composition, information distribution, and analysis coordination for visual data analysis in multi-display environments. In particular, a basic model for layout modeling is introduced, a graphical interface for interacti…
This work aims to investigate how the amenities offered by Intelligent Environments can be used to shape new types of useful, exciting and fulfilling experiences while watching sports or movies. Towards this direction, two ambient media players were developed aspiring to offer live access to secondary information via the available displays of an Intelligent Living Room, and to appropriately exploit the technological equipment so as to support natural interaction. Expert-based evaluation experiments revealed some factors that can influence the overall experience significantly, without hindering…
Eye gaze controlled interfaces allow us to directly manipulate a graphical user interface just by looking at it. This technology has great potential in military aviation, in particular, operating different displays in situations where pilots hands are occupied with flying the aircraft. This paper reports studies on analyzing accuracy of eye gaze controlled interface inside aircraft undertaking representative flying missions. We reported that pilots can undertake representative pointing and selection tasks at less than 2 secs on average. Further, we evaluated the accuracy of eye gaze tracking g…
Haptic devices have been employed to immerse users in VR environments. In particular, hand and finger haptic devices have been deeply developed. However, this type of device occlude the hand detection by some tracking systems, or in other tracking systems, it is uncomfortable for the users to wear two-hand devices (haptic and tracking device). We introduce RecyGlide, which is a novel wearable forearm multimodal display at the forearm. The RecyGlide is composed of inverted five-bar linkages and vibration motors. The device provides multimodal tactile feedback such as slippage, a force vector, p…
Efficient motion intent communication is necessary for safe and collaborative work environments with collocated humans and robots. Humans efficiently communicate their motion intent to other humans through gestures, gaze, and social cues. However, robots often have difficulty efficiently communicating their motion intent to humans via these methods. Many existing methods for robot motion intent communication rely on 2D displays, which require the human to continually pause their work and check a visualization. We propose a mixed reality head-mounted display visualization of the proposed robot …
As computers become more ubiquitous, traditional two-dimensional interfaces must be replaced with interfaces based on a three-dimensional metaphor. However, these interfaces must still be as simple and functional as their two-dimensional predecessors. This paper introduces SWiM, a new interface for moving application windows between various screens, such as wall displays, laptop monitors, and desktop displays, in a three-dimensional physical environment. SWiM was designed based on the results of initial "paper and pencil" user tests of three possible interfaces. The results of these tests led …
Virtual and Augmented Reality have the potential to change information work. The ability to modify the workers senses can transform everyday environments into a productive office, using portable head-mounted displays combined with conventional interaction devices, such as keyboards and tablets. While a stream of better, cheaper and lighter HMDs have been introduced for consumers in recent years, there are still many challenges to be addressed to allow this vision to become reality. This chapter summarizes the state of the art in the field of extended reality for knowledge work in everyday envi…
Recent advancements in mobile devices encourage researchers to utilize them in collaborative environments as a medium to interact with large shared wall-displays. In this paper, we focus on a semi-controlled user study that we conducted to measure the collaborative coupling ratio between partners working in pairs in a collaborative setup equipped with a shared tiled-wall display and multiple mobile devices. We invited 36 participants in 18 pairs to take part in our experiment in order to analyze how they communicate and collaborate with each other during the experiment. We observed their colla…
Head gesture is a natural means of face-to-face communication between people but the recognition of head gestures in the context of virtual reality and use of head gesture as an interface for interacting with virtual avatars and virtual environments have been rarely investigated. In the current study, we present an approach for real-time head gesture recognition on head-mounted displays using Cascaded Hidden Markov Models. We conducted two experiments to evaluate our proposed approach. In experiment 1, we trained the Cascaded Hidden Markov Models and assessed the offline classification perform…
In this paper, we present a multi-user interaction interface for a large immersive space that supports simultaneous screen interactions by combining (1) user input via personal smartphones and Bluetooth microphones, (2) spatial tracking via an overhead array of Kinect sensors, and (3) WebSocket interfaces to a webpage running on the large screen. Users are automatically, dynamically assigned personal and shared screen sub-spaces based on their tracked location with respect to the screen, and use a webpage on their personal smartphone for touchpad-type input. We report user experiments using ou…
Optical see-though head-mounted displays (OST HMDs) are one of the key technologies for merging virtual objects and physical scenes to provide an immersive mixed reality (MR) environment to its user. A fundamental limitation of HMDs is, that the user itself cannot be augmented conveniently as, in casual posture, only the distal upper extremities are within the field of view of the HMD. Consequently, most MR applications that are centered around the user, such as virtual dressing rooms or learning of body movements, cannot be realized with HMDs. In this paper, we propose a novel concept and pro…
The information provided to a person's visual system by extended reality (XR) displays is not a veridical match to the information provided by the real world. Due in part to graphical limitations in XR head-mounted displays (HMDs), which vary by device, our perception of space may be altered. However, we do not yet know which properties of virtual objects rendered by HMDs -- particularly augmented reality displays -- influence our ability to understand space. In the current research, we evaluate how immersive graphics affect spatial perception across three unique XR displays: virtual reality (…
In information-rich environments, the competition for users' attention leads to a flood of content from which people often find hard to sort out the most relevant and useful pieces. Using Twitter as a case study, we applied an attention economy solution to generate the most informative tweets for its users. By considering the novelty and popularity of tweets as objective measures of their relevance and utility, we used the Huberman-Wu algorithm to automatically select the ones that will receive the most attention in the next time interval. Their predicted popularity was confirmed by using Twit…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
This paper presents a novel haptic device DeltaFinger designed to deliver the force of interaction with virtual objects by guiding user's finger with wearable delta mechanism. The developed interface is capable to deliver 3D force vector to the fingertip of the index finger of the user, allowing complex rendering of virtual reality (VR) environment. The developed device is able to produce the kinesthetic feedback up to 1.8 N in vertical projection and 0.9 N in horizontal projection without restricting the motion freedom of of the remaining fingers. The experimental results showed a sufficient …
With recent advances of Virtual Reality (VR) technology, the deployment of such will dramatically increase in non-entertainment environments, such as professional education and training, manufacturing, service, or low frequency/high risk scenarios. Clinical education is an area that especially stands to benefit from VR technology due to the complexity, high cost, and difficult logistics. The effectiveness of the deployment of VR systems, is subject to factors that may not be necessarily considered for devices targeting the entertainment market. In this work, we systematically compare a wide ra…
Unmanned Aerial Vehicles (UAVs) have moved beyond a platform for hobbyists to enable environmental monitoring, journalism, film industry, search and rescue, package delivery, and entertainment. This paper describes 3D displays using swarms of flying light specks, FLSs. An FLS is a small (hundreds of micrometers in size) UAV with one or more light sources to generate different colors and textures with adjustable brightness. A synchronized swarm of FLSs renders an illumination in a pre-specified 3D volume, an FLS display. An FLS display provides true depth, enabling a user to perceive a scene mo…
