In-place gesture-based virtual locomotion techniques enable users to control their viewpoint and intuitively move in the 3D virtual environment. A key research problem is to accurately and quickly recognize in-place gestures, since they can trigger specific movements of virtual viewpoints and enhance user experience. However, to achieve real-time experience, only short-term sensor sequence data (up to about 300ms, 6 to 10 frames) can be taken as input, which actually affects the classification performance due to limited spatio-temporal information. In this paper, we propose a novel long-term memory augmented network for in-place gestures classification. It takes as input both short-term gesture sequence samples and their corresponding long-term sequence samples that provide extra relevant spatio-temporal information in the training phase. We store long-term sequence features with an external memory queue. In addition, we design a memory augmented loss to help cluster features of the same class and push apart features from different classes, thus enabling our memory queue to memorize more relevant long-term sequence features. In the inference phase, we input only short-term sequence samples to recall the stored features accordingly, and fuse them together to predict the gesture class. We create a large-scale in-place gestures dataset from 25 participants with 11 gestures. Our method achieves a promising accuracy of 95.1% with a latency of 192ms, and an accuracy of 97.3% with a latency of 312ms, and is demonstrated to be superior to recent in-place gesture classification techniques. User study also validates our approach. Our source code and dataset will be made available to the community.
Enabling Intuitive Human-Robot Teaming Using Augmented Reality and Gesture Control
US CCDC Army Research Laboratory（美国CCDC陆军研究实验室）；Oak Ridge Associated Universities（橡树岭联合大学）；Defence Science and Technology（国防科学技术集团）；Swinburne University of Technology（斯威本科技大学）
OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition
School of Systems Science and Engineering, Sun Yat-sen University（中山大学系统科学与工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）；Nankai University（南开大学）；Defense Innovation Institute, Academy of Military Sciences（军事科学院国防创新研究院）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）
新技术使普通人能够以负担得起的价格在家中访问虚拟现实。与沉浸式虚拟现实交互时最重要的任务之一是在虚拟环境（VE）中导航。可以说，实现这一点的最佳方法是使用直接控制接口。其中，自然行走（NW）提供了令人愉快的用户体验。然而，在VE中支持直接控制接口的常见技术具有限制，使得难以在狭窄的家庭环境中使用这些方法。事实上，西北需要畅通无阻的开放空间。为了解决这个问题，我们提出了一种新的虚拟运动技术，组合步行到位（CNOW）。CNET允许人们利用可用的物理空间，并使他们能够使用NW在虚拟世界中导航。对于更长的距离，我们采用原地行走（Walking in Place，简称WARE），使他们能够在虚拟世界中移动，而不仅仅局限于狭窄的现实房间。然而，在沉浸式交替现实中漫游，而在混乱的环境中移动可能会导致人们绊倒和摔倒。为了解决这些问题，我们开发了增强虚拟现实（AVR），告知用户现实世界的危险，如椅子，抽屉，墙壁通过代理和标志放置在虚拟世界中。因此，我们建议CWIP-AVR作为一种在自己家中狭窄的范围内安全探索VR的方式。据我们所知，这是第一种以安全的方式组合不同运动方式的方法。我们在一项有20名参与者的用户研究中对它进行了评估，以验证他们在狭窄和混乱的现实空间中行走时导航虚拟世界的能力。我们的研究结果表明，CWIP-AVR允许人们安全地导航VR，在运动模式之间灵活切换，同时保持良好的沉浸感。
