arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.02441cs.MMcs.AI
收藏

LLMER:利用大语言模型生成的JSON数据构建交互式扩展现实世界

LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models

Pennsylvania State University(宾夕法尼亚州立大学);George Washington University(乔治华盛顿大学)

Jiangong Chen, Xiaoyi Wu, Tian Lan, Bin Li

AI总结:

针对大语言模型与XR结合时存在的参数提取难、生成错误多、脚本易崩溃等问题,提出LLMER框架,通过将自然语言转为JSON数据、采用多阶段策略,大幅降低token消耗与任务耗时,提升XR世界生成的稳定性与效率。

摘要:

将GPT-4等大语言模型(Large Language Models, LLMs)与扩展现实(Extended Reality, XR)技术相结合,有望构建真正沉浸式的XR环境,通过自然语言与人类用户交互,例如从音频输入生成3D场景并为其添加动画效果。然而,XR环境的复杂性使得研究人员难以从海量的XR素材中准确提取相关的上下文数据和场景/对象参数。这不仅会导致按使用量付费的模型成本上升,还会增加生成错误的概率。此外,现有的侧重于代码脚本生成的方法往往容易出现生成错误,导致脚本存在缺陷或无效、应用程序崩溃,最终降低用户体验。为克服这些挑战,我们提出了LLMER,这是一种利用LLMs生成的JSON数据创建交互式XR世界的新型框架。与以往侧重于代码脚本生成的方法不同,LLMER将自然语言输入转换为JSON数据,显著降低了应用程序崩溃的可能性和处理延迟。它采用多阶段策略,仅提供适配用户请求的必要上下文信息,并配备了专为各类XR任务设计的多个模块。我们的初步用户研究证明了该系统的有效性:与当前最优方法相比,其消耗的token减少了80%以上,任务完成时间缩短了约60%。对用户反馈的分析也为进一步优化指明了一系列方向。

英文摘要:

The integration of Large Language Models (LLMs) like GPT-4 with Extended Reality (XR) technologies offers the potential to build truly immersive XR environments that interact with human users through natural language, e.g., generating and animating 3D scenes from audio inputs. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. It leads to not only increased costs with pay-per-use models, but also elevated levels of generation errors. Moreover, existing approaches focusing on coding script generation are often prone to generation errors, resulting in flawed or invalid scripts, application crashes, and ultimately a degraded user experience. To overcome these challenges, we introduce LLMER, a novel framework that creates interactive XR worlds using JSON data generated by LLMs. Unlike prior approaches focusing on coding script generation, LLMER translates natural language inputs into JSON data, significantly reducing the likelihood of application crashes and processing latency. It employs a multi-stage strategy to supply only the essential contextual information adapted to the user's request and features multiple modules designed for various XR tasks. Our preliminary user study reveals the effectiveness of the proposed system, with over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to state-of-the-art approaches. The analysis of users' feedback also illuminates a series of directions for further optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑