arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.09536cs.HCcs.MM
收藏

动态多模态表达生成用于基于大语言模型的教育代理:从用户体验视角

Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective

Ninghao Wan, Jiarun Song, Fuzheng Yang

AI总结:

本文提出基于大语言模型的动态多模态表达生成方法,通过生成协调的语音和手势指令,提升教育代理的沉浸感和自然表达能力。

摘要:

在虚拟现实(VR)教育场景中,教育代理(PAs)通过逼真的外观和交互行为增强沉浸式学习。然而,大多数现有PAs依赖静态语音和简单手势。这种限制减少了它们动态适应教学内容语义上下文的能力。因此,在教学过程中,互动往往缺乏自然性和有效性。为了解决这一挑战,本研究提出了一种基于大语言模型(LLM)的多模态表达生成方法,通过构建语义敏感的提示来生成协调的语音和手势指令,使教学语义与多模态表达行为实现动态对齐。通过以用户体验为导向的主观实验开发并评估了基于VR的PA原型。结果表明,动态生成的多模态表达显著增强了学习者的感知学习效果、参与度和使用意愿,同时有效缓解了学习过程中的疲劳和无聊感。此外,语音和手势的结合动态表达显著增强了学习者对人类化和社交存在的感知。这些发现为构建更沉浸和自然表达的智能PAs提供了新的见解和设计指南。

英文摘要:

In virtual reality (VR) educational scenarios, Pedagogical agents (PAs) enhance immersive learning through realistic appearances and interactive behaviors. However, most existing PAs rely on static speech and simple gestures. This limitation reduces their ability to dynamically adapt to the semantic context of instructional content. As a result, interactions often lack naturalness and effectiveness in the teaching process. To address this challenge, this study proposes a large language model (LLM)-driven multimodal expression generation method that constructs semantically sensitive prompts to generate coordinated speech and gesture instructions, enabling dynamic alignment between instructional semantics and multimodal expressive behaviors. A VR-based PA prototype was developed and evaluated through user experience-oriented subjective experiments. Results indicate that dynamically generated multimodal expressions significantly enhance learners' perceived learning effectiveness, engagement, and intention to use, while effectively alleviating feelings of fatigue and boredom during the learning process. Furthermore, the combined dynamic expression of speech and gestures notably enhances learners' perceptions of human-likeness and social presence. The findings provide new insights and design guidelines for building more immersive and naturally expressive intelligent PAs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑