声音在增强现实(AR)中对于提升用户体验和沉浸感起着至关重要的作用。然而,当前平台缺乏对AR声音创作的支持,原因在于交互类型有限、收集和指定上下文信息存在挑战,以及难以获取匹配的声音素材。我们提出了SonifyAR,一个基于大语言模型(LLM)的AR声音创作系统,能够为AR体验生成上下文感知的音效。SonifyAR扩展了当前AR声音的设计空间,并实现了一个演示编程(Programming by Demonstration,PbD)流水线,以自动收集AR事件的上下文信息,包括虚拟内容语义和真实世界上下文。随后,这些上下文信息由大语言模型处理,通过推荐、检索、生成和迁移四种方法获取音效。为了评估我们系统的可用性和性能,我们开展了一项有八名参与者参加的用户研究,并创建了五个示例应用,包括一个基于AR的科学实验、一个改善AR头戴设备安全性的案例,以及一个面向低视力AR用户的辅助示例。
英文摘要:
Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual content semantics and real world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, an improving case for AR headset safety, and an assisting example for low vision AR users.
MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR
Google Mountain View CA USA(谷歌山景城);University of Michigan Ann Arbor MI USA(密歇根大学安 Arbor分校);Columbia University New York NY USA(哥伦比亚大学);Google San Francisco CA USA(谷歌旧金山)
Towards Ubiquitous Semantic Metaverse: Challenges, Approaches, and Opportunities
Real-Time and Embedded Computing Systems Research Centre (CISTER)(实时与嵌入式计算系统研究中心(CISTER));CyLab Security and Privacy Institute, Carnegie Mellon University(卡内基梅隆大学CyLab安全与隐私研究所);Singapore University of Technology and Design(新加坡科技设计大学);Commonwealth Scientific and Industrial Research Organization (CSIRO)(联邦科学与工业研究组织(CSIRO));Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学(MBZUAI));Nanyang Technological University(南洋理工大学)