arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.09176cs.HC
收藏

LIVE-GS:通过大语言模型实现交互式VR体验的物理感知高斯点云

LIVE-GS: LLM Powers Interactive VR Experience with Physics-Aware Gaussian Splatting

Shanghai Jiao Tong University(上海交通大学);Zhejiang University of Science and Technology(浙江科技学院);Shanghai AI Lab(上海人工智能实验室)

Haotian Mao, Hangyu Zhou, Zhuoxiong Xu, Siyue Wei, Yule Quan, Yan Zhang, Zixuan Guo, Nianchen Deng, Xubo Yang

AI总结:

LIVE-GS利用大语言模型快速生成动态高斯点云资产并实现实时VR交互,通过物理属性分析提升虚拟现实中的物理交互真实感。

摘要:

随着3D高斯点云(3DGS)成为新型视图合成和场景重建的领先方法,其在数字资产创建中的潜力引起了广泛关注。越来越多基于GS的资产库正在建立。然而,生成基于物理的动态资产仍是一个耗时且需要专业知识的任务,尤其是对非专业人士而言。在本文中,我们提出了LIVE-GS,一个由大语言模型(LLMs)驱动的高现实感虚拟现实(VR)系统,能够快速创建动态高斯点云资产并实现实时VR交互。为了指导我们的系统设计,我们进行了访谈,以考察当前基于GS的VR系统所面临的挑战以及用户的具体需求。基于这些见解,我们利用GPT-4o分析影响用户交互的关键物理属性,确保VR中的物理交互符合现实现象。LIVE-GS的关键创新在于其能够在10秒内从静态高斯点云资产中预测出合理的参数,同时保持高质量的VR交互。为了验证我们的方法,我们邀请了在物理模拟方面有经验的参与者手动调整物理参数,为资产质量和创作效率提供了比较基准。我们还进行了全面的用户研究,以评估系统可用性和用户满意度。实验结果表明,LIVE-GS通过利用LLMs的场景理解能力,能够实现高效的物理场景创建和自然的交互,而无需手动设计或注释。

英文摘要:

As 3D Gaussian Splatting (3DGS) emerges as a leading approach for novel view synthesis and scene reconstruction, its potential in digital asset creation has gained significant attention. An increasing number of asset libraries based on GS are being established. However, generating physics-based dynamic assets remains a time-consuming and expertise-intensive task, especially for non-experts. In this paper, we propose LIVE-GS, a highly realistic Virtual Reality (VR) system powered by Large Language Models (LLMs), which enables rapid creation of dynamic Gaussian assets and real-time VR interactions. To inform our system design, we conducted interviews to examine challenges faced by current GS-based VR systems and the specific demands of users. Based on these insights, we employed GPT-4o to analyze key physical properties of objects that significantly impact user interactions, ensuring physics-based interactions in VR align with real-world phenomena. A key innovation of LIVE-GS is its ability to predict reasonable parameters in just 10 seconds from static Gaussian assets while maintaining high-quality VR interactions. To validate our approach, we invited participants experienced in physical simulation to manually adjust physical parameters, providing a baseline for comparison in both asset quality and authoring efficiency. We also conducted a comprehensive user study to evaluate system usability and user satisfaction. Experimental results demonstrate that LIVE-GS, leveraging LLMs' scene understanding capabilities, can achieve efficient physical scene creation and natural interactions without requiring manual design or annotation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑