The advent of immersive Virtual Reality applications has transformed various domains, yet their integration with advanced artificial intelligence technologies like Visual Language Models remains underexplored. This study introduces a pioneering approach utilizing VLMs within VR environments to enhance user interaction and task efficiency. Leveraging the Unity engine and a custom-developed VLM, our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions. The incorporation of speech-to-text and text-to-speech technologies allows for seamless communication between the user and the VLM, enabling the system to guide users through complex tasks effectively. Preliminary experimental results indicate that utilizing VLMs not only reduces task completion times but also improves user comfort and task engagement compared to traditional VR interaction methods.
Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning
Technische Universität Darmstadt(达姆施塔特工业大学);Center for Mind, Brain and Behavior, Uni. Marburg and JLU Giessen(马尔堡大学与吉森尤斯图斯·李比希大学心智、大脑与行为研究中心);Amazon Alexa(亚马逊Alexa)
Hybrid LLM-based Intelligent Framework for Robot Task Scheduling
Department of Computer Science and Engineering, Mississippi State University(密苏里州立大学计算机科学与工程系);Graduate School of Arts and Sciences, Columbia University(哥伦比亚大学研究生院);Consumer and Community Banking, JPMorgan Chase(摩根大通消费与社区银行业);Department of Data Science, Columbia University(哥伦比亚大学数据科学系);Department of Electrical Engineering, Columbia University(哥伦比亚大学电气工程系)
Intelligent Materials Modelling: Large Language Models Versus Partial Least Squares Regression for Predicting Polysulfone Membrane Mechanical Performance
Jailbreaking Large Vision Language Models in Intelligent Transportation Systems
KFSCIS, Florida International University(凯斯-西储大学信息科学学院);Knight Foundation School of Computing and Information Sciences(骑士基金会计算与信息科学学院);Learning for InterDependent Networks Laboratory (solid lab)(依赖网络学习实验室)
Predicting Learning Performance with Large Language Models: A Study in Adult Literacy
Institute for Intelligent Systems, University of Memphis(孟菲斯大学智能系统研究所);University of Memphis(孟菲斯大学);Human-Computer Interaction Institute, Carnegie Mellon University(卡内基梅隆大学人机交互研究所);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Polytechnic University(香港理工大学)
SENSAR: A Visual Tool for Intelligent Robots for Collaborative Human-Robot Interaction
在智能机器人和人类之间建立共同点需要将机器人的意图、行为和知识传达给人类,以建立信任并确保共享环境中的安全。本文介绍了SENSAR(Seeing Everything iN Situ with增强现实),一个增强现实机器人系统,使机器人能够在真实世界的背景下与渲染的图形交流他们的感官和认知数据,允许用户理解,纠正和验证机器人对世界的感知。我们的系统旨在通过建立人类和机器人的感知对齐的共同点来支持人机交互研究。
How to Integrate Digital Twin and Virtual Reality in Robotics Systems? Design and Implementation for Providing Robotics Maintenance Services in Data Centers
University of Twente(特文特大学);Beijing Hanning Tech Co., Ltd(北京汉宁科技有限公司)