3D Augmented Reality Tangible User Interface using Commodity Hardware
Dimitrios Chamzas, Konstantinos Moustakas
备注：
15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (GRAPP), Valletta, Malta, 25-27 February 2020
During the last years, the emerging field of Augmented and Virtual Reality (AR-VR) has seen tremendous growth. An interface that has also become very popular for the AR systems is the tangible interface or passive-haptic interface. Specifically, an interface where users can manipulate digital information with input devices that are physical objects. This work presents a low cost Augmented Reality system with a tangible interface that offers interaction between the real and the virtual world. The system estimates in real-time the 3D position of a small colored ball (input device), it maps it to the 3D virtual world and then uses it to control the AR application that runs in a mobile device. Using the 3D position of our "input" device, it allows us to implement more complicated interactivity compared to a 2D input device. Finally, we present a simple, fast and robust algorithm that can estimate the corners of a convex quadrangle. The proposed algorithm is suitable for the fast registration of markers and significantly improves performance compared to the state of the art.
IIIT-Delhi（德里英迪拉甘地信息技术学院）；University of Calgary（卡尔加里大学）
This paper introduces Teachable Reality, an augmented reality (AR) prototyping tool for creating interactive tangible AR applications with arbitrary everyday objects. Teachable Reality leverages vision-based interactive machine teaching (e.g., Teachable Machine), which captures real-world interactions for AR prototyping. It identifies the user-defined tangible and gestural interactions using an on-demand computer vision model. Based on this, the user can easily create functional AR prototypes without programming, enabled by a trigger-action authoring interface. Therefore, our approach allows t…
Post-Lecture Interactive Environments for Conceptual Learning: A Randomized Comparison of Mixed Reality and Tangible Instruction in Undergraduate STEM Education
Exploring and manipulating complex virtual objects is challenging due to limitations of conventional controllers and free-hand interaction techniques. We present the TanGi toolkit which enables novices to rapidly build physical proxy objects using Composable Shape Primitives. TanGi also provides Manipulators allowing users to build objects including movable parts, making them suitable for rich object exploration and manipulation in VR. With a set of different use cases and applications we show the capabilities of the TanGi toolkit, and evaluate its use. In a study with 16 participants, we demo…
In gaming, customizing individual characters, can create personal bonds between players and their characters. Hence, character customization is a standard component in many games. While mobile Augmented Reality (AR) games become popular, to date, no 3D character editor for AR games exists. We investigate the feasibility of 3D character customization for smartphone-based AR in an iterative design process. Specifically, we present findings from creating AR prototypes in a handheld AR setting. In a first user study, we found that a tangible AR prototype resulted in higher hedonistic measures than…
Mixed reality head-mounted displays (mxdR-HMD) have the potential to visualize volumetric medical imaging data in holograms to provide a true sense of volumetric depth. An effective user interface, however, has yet to be thoroughly studied. Tangible user interfaces (TUIs) enable a tactile interaction with a hologram through an object. The object has physical properties indicating how it might be used with multiple degrees-of-freedom. We propose a TUI using a planar object (PO) for the holographic medical volume visualization and exploration. We refer to it as mxdR hologram slicer (mxdR-HS). Us…
Going Down the Abstraction Stream with Augmented Reality and Tangible Robots: the Case of Vector Instruction
CHILI Lab Ecole Polytechnique Fédérale de Lausanne（CHILI实验室 雅各布·冯·利希滕堡联邦理工学院）；School of Computing & Information Systems University of Melbourne（计算与信息系统学院 澳大利亚墨尔本大学）
The University of Tokyo（东京大学）；University of Calgary（卡尔加里大学）；IIIT-Delhi（德里印度理工学院）；University of Chicago（芝加哥大学）
This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bidirectional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to "affect" each other through physical actuation and digital computation. In the existing AR sketching, the relationship between virtual and physical worlds is only one-directional -- while physical interaction can affect virtual sketches, virtual sketches have no return effect on the physical objects or environment. In contrast, bi-directional sketching interac…
Digital devices are now ubiquitous and have the potential to be used to support positive changes in human lives and promote psychological well-being. This paper presents three interactive systems that we created focusing on introspection activities, leveraging tangible interaction and spatial augmented reality. More specifically, we describe anthropomorphic augmented avatars that display the users' inner states using physiological sensors. We also present a first prototype of an augmented sandbox specifically dedicated to promoting mindfulness activities.
An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual reality (AR/VR) are capable of finger and hand tracking. We report on the accuracy of off-the-shelf VR and AR HMDs when used for touch-based tasks such as pointing or drawing. Specifically, we report on the finger tracking accuracy of the VR head-mounted displays Oculus Quest, Vive Pro and the Leap Motion controller, when attached to a VR HMD, as well as the finger tracking accuracy of the AR head-mounted displays Microsoft HoloLens 2 and Magic Leap. We present the results of two experiments in whic…
This chapter gives an overview of interaction techniques for mixed reality including augmented and virtual reality (AR/VR). Various modalities for input and output are discussed. Specifically, techniques for tangible and surface-based interaction, gesture-based, pen-based, gaze-based, keyboard and mouse-based, as well as haptic interaction are discussed. Furthermore, the combination of multiple modalities in multisensory and multimodal interaction, as well as interaction using multiple physical or virtual displays, are presented. Finally, interaction with intelligent virtual agents is consider…
Digital interaction with everyday objects has become popular since the proliferation of camera-based systems that detect and augment objects "just-in-time". Common systems use a vision-based approach to detect objects and display their functionalities to the user. Sensors, such as color and depth cameras, have become inexpensive and allow seamless environmental tracking in mobile as well as stationary settings. However, object detection in different contexts faces challenges as it highly depends on environmental parameters and the conditions of the object itself. In this work, we present three…
We propose a new approach for interaction in Virtual Reality (VR) using mobile robots as proxies for haptic feedback. This approach allows VR users to have the experience of sharing and manipulating tangible physical objects with remote collaborators. Because participants do not directly observe the robotic proxies, the mapping between them and the virtual objects is not required to be direct. In this paper, we describe our implementation, various scenarios for interaction, and a preliminary user study.
This paper proposes the concept of "live-action virtual reality games" as a new genre of digital games based on an innovative combination of live-action, mixed-reality, context-awareness, and interaction paradigms that comprise tangible objects, context-aware input devices, and embedded/embodied interactions. Live-action virtual reality games are "live-action games" because a player physically acts out (using his/her real body and senses) his/her "avatar" (his/her virtual representation) in the game stage, which is the mixed-reality environment where the game happens. The game stage is a kind …
The last two decades have seen the emergence and steady development of tangible user interfaces. While most of these interfaces are applied for input - with output still on traditional computer screens - the goal of programmable matter and actuated shape-changing materials is to directly use the physical objects for visual or tangible feedback. Advances in material sciences and flexible display technologies are investigated to enable such reconfigurable physical objects. While existing solutions aim for making physical objects more controllable via the digital world, we propose an approach whe…
The author considers a class of mechatronic puzzles falling in the mixed-reality category, present examples of such devices, and propose a way to categorize them. Close relationships of such devices with the Tangible User Interface are described. The device designed by the author as an illustration of a mixed reality puzzle is presented.
Towards Tangible Immersion for Cobot Programming-by-Demonstration: Visual, Tactile and Haptic Interfaces for Mixed-Reality Cobot Automation in Semiconductor Manufacturing
Intel Labs, Intelligent System Research, Human-Robot Collaboration（英特尔实验室、智能系统研究、人机协作）
This article reports prototype web apps that use commodity, open-source technologies for augmented and virtual reality to provide immersive, interactive human-computer interfaces for chemistry, structural biology and related disciplines. The examples, which run in any standard web browser and are accessible at https://lucianoabriata.altervista.org/jsinscience/arjs/armodeling/ together with demo videos, showcase applications that could go well beyond pedagogy, i.e. advancing actual utility in research settings: molecular visualization at atomistic and coarse-grained levels in interactive immers…
We contribute MolecuSense, a virtual version of a physical molecule construction kit, based on visualization in Virtual Reality (VR) and interaction with force-feedback gloves. Targeting at chemistry education, our goal is to make virtual molecule structures more tangible. Results of an initial user study indicate that the VR molecular construction kit was positively received. Compared to a physical construction kit, the VR molecular construction kit is on the same level in terms of natural interaction. Besides, it fosters the typical digital advantages though, such as saving, exporting, and s…
This paper presents a design space of interaction techniques to engage with visualizations that are printed on paper and augmented through Augmented Reality. Paper sheets are widely used to deploy visualizations and provide a rich set of tangible affordances for interactions, such as touch, folding, tilting, or stacking. At the same time, augmented reality can dynamically update visualization content to provide commands such as pan, zoom, filter, or detail on demand. This paper is the first to provide a structured approach to mapping possible actions with the paper to interaction commands. Thi…
We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational w…
We implemented a system able to locate people indoor, with the purpose of providing assistive services. Such approach is particularly important for the Art, for providing information on exhibitions, art galleries and museums, and to allow the access to the cultural heritage patrimony to people with disabilities. The system may provide also very important information and input to elderly people, helping them to perceive more deeply the reality and the beauty of art. The system is based on Beacons, very small and low power consumption devices, and Human Body Communication protocols. The Beacons,…
Advancements in heterogeneous computing technologies enable the significant potential of virtual reality (VR) applications. To offer the best user experience (UX), a system should adopt an untethered, wireless-network-based architecture to transfer VR content between the user and the content generator. However, modern wireless network technologies make implementing such an architecture challenging, as VR applications require superior video quality -- with high resolution, high frame rates, and very low latency. This paper presents OpenUVR, an open-source framework that uses commodity hardware …
Virtual reality (VR) offers immersive visualization and intuitive interaction. We leverage VR to enable any biomedical professional to deploy a deep learning (DL) model for image classification. While DL models can be powerful tools for data analysis, they are also challenging to understand and develop. To make deep learning more accessible and intuitive, we have built a virtual reality-based DL development environment. Within our environment, the user can move tangible objects to construct a neural network only using their hands. Our software automatically translates these configurations into…
Our sense of touch is essential and permeates in interactions involving natural explorations and affective communications. For instance, we routinely judge the ripeness of fruit at the grocery store, caress the arm of a spouse to offer comfort, and stroke textiles to gauge their softness. Meanwhile, interactive displays that provide tactile feedback are becoming normal and ubiquitous in our daily lives, and are extending rich and immersive interactions into augmented and virtual reality. To replicate touch sensation and make virtual objects feel tangible, such feedback will need to relay a sen…
Manipulating Tangible Virtual Object Dynamics to Promote Learning of Precision Force Generation
Department of Cognitive Robotics, Delft University of Technology, Delft, 2628 CD, The Netherlands；Department of Socio-medical Sciences, Erasmus School of Health Policy \& Management, Erasmus University Rotterdam, Rotterdam, 3062PA, The Netherlands.；Department of Applied Mathematics, Delft University of Technology, Delft, 2628 CD, The Netherlands；Department of Rehabilitation Medicine, Erasmus Medical Center, Rotterdam, 3015 GD, The Netherlands
Wireless Sensor Networks (WSNs) often lack interfaces for remote debugging. Thus, fault diagnosis and troubleshooting are conducted at the deployment site. Currently, WSN operators lack dedicated tools that aid them in this process. Therefore, we introduce EyeSec, a tool for WSN monitoring and maintenance in the field. An Augmented Reality Device (AR Device) identifies sensor nodes using optical markers. Portable Sniffer Units capture network traffic and extract information. With those data, the AR Device network topology and data flows between sensor nodes are visualized. Unlike previous tool…
We present Bionic Tracking, a novel method for solving biological cell tracking problems with eye tracking in virtual reality using commodity hardware. Using gaze data, and especially smooth pursuit eye movements, we are able to track cells in time series of 3D volumetric datasets. The problem of tracking cells is ubiquitous in developmental biology, where large volumetric microscopy datasets are acquired on a daily basis, often comprising hundreds or thousands of time points that span hours or days. The image data, however, is only a means to an end, and scientists are often interested in the…
We describe a framework for interactive molecular dynamics in a multiuser virtual reality environment, combining rigorous cloud-mounted physical atomistic simulation with commodity virtual reality hardware, which we have made accessible to readers (see isci.itch.io/nsb-imd). It allows users to visualize and sample, with atomic-level precision, the structures and dynamics of complex molecular structures 'on the fly', and to interact with other users in the same virtual environment. A series of controlled studies, wherein participants were tasked with a range of molecular manipulation goals (thr…
Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（Skoltech））
TouchVR is a novel wearable haptic interface which can deliver multimodal tactile stimuli on the palm by DeltaTouch haptic display and vibrotactile feedback on the fingertips by vibration motors for the Virtual Reality (VR) user. DeltaTouch display is capable of generating 3D force vector at the contact point and presenting multimodal tactile sensation of weight, slippage, encounter, softness, and texture. The VR system consists of HTC Vive Pro base stations and head-mounted display (HMD), and Leap Motion controller for tracking the user's hands motion in VR. The MatrixTouch, BallFeel, and Rob…
In recent years, commodity mobile devices equipped with cameras and inertial measurement units (IMUs) have attracted much research and design effort for augmented reality (AR) and robotics applications. Based on such sensors, many commercial AR toolkits and public benchmark datasets have been made available to accelerate hatching and validating new ideas. To lower the difficulty and enhance the flexibility in accessing the rich raw data of typical AR sensors on mobile devices, this paper present the mobile AR sensor (MARS) logger for two of the most popular mobile operating systems, Android an…
