arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09649cs.HCcs.CL
收藏

ReactGenie:一个使用大语言模型进行复杂多模态交互的开发框架

ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models

Stanford University(斯坦福大学);Tsinghua University(清华大学);Northeastern University(东北大学)

Jackie Junrui Yang, Yingtian Shi, Yuhan Zhang, Karina Li, Daniel Wan Rosli, Anisha Jain, Shuning Zhang, Tianshi Li, James A. Landay, Monica S. Lam

AI总结:

ReactGenie是一个基于大语言模型的多模态交互开发框架,通过将用户语音和触控命令解析为NLPL并组合计算原语,使开发者能快速构建支持丰富复杂命令的多模态应用,同时提升最终用户的任务完成效率和体验。

摘要:

通过结合语音和触控交互,多模态界面可以超越单一模态的效率。传统的多模态框架需要开发者付出繁重的劳动来支持丰富的多模态命令,其中用户的多模态命令可能涉及动作/函数调用的指数级组合。本文提出了ReactGenie,一个编程框架,它更好地将多模态输入与计算模型分离,使开发者能够轻松创建高效且功能强大的多模态界面。ReactGenie使用基于大语言模型的神经语义解析器,将多模态用户命令翻译成NLPL(自然语言编程语言),这是我们创建的一种编程语言。ReactGenie运行时解释解析后的NLPL,并组合计算模型中的原语来实现复杂的用户命令。因此,ReactGenie允许轻松实现多模态应用最终用户前所未有的命令丰富性。我们的评估表明,12名开发者平均可以在2.5小时内学习并构建一个非平凡的ReactGenie应用。此外,与传统GUI相比,最终用户使用ReactGenie应用可以更快地完成任务,并且任务负载更低。

英文摘要:

By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. Traditional multimodal frameworks require laborious developer work to support rich multimodal commands where the user's multimodal command involves possibly exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework that better separates multimodal input from the computational model to enable developers to create efficient and capable multimodal interfaces with ease. ReactGenie translates multimodal user commands into NLPL (Natural Language Programming Language), a programming language we created, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed NLPL and composes primitives in the computational model to implement complex user commands. As a result, ReactGenie allows easy implementation and unprecedented richness in commands for end-users of multimodal apps. Our evaluation showed that 12 developers can learn and build a nontrivial ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end-users can complete tasks faster and with less task load using ReactGenie apps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑