用于话轮转换预测的多模态Transformer模型：合作游戏过程中其对人机智能体交互会话动态的影响
Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay
AI总结：
本研究提出融合多模态信息的跨模态Transformer话轮转换预测模型，在合作游戏场景的用户研究中验证其可提升人机对话流畅自然度，维持平衡会话动态。
摘要：
本研究探究人机智能体交互（HAI）场景下的多模态话轮转换预测，重点聚焦合作游戏环境。研究包含模型开发与后续用户研究两部分，旨在深化我们对口语对话系统（SDS）的认知并优化其会话动态。\n在建模阶段，我们提出了一种基于Transformer的新型深度学习（DL）模型，该模型可同时整合文本、视觉、音频以及游戏内上下文数据等多种模态，实时预测话轮转换事件。模型采用跨模态Transformer（Crossmodal Transformer）架构，可有效融合来自这些不同模态的信息，实现更全面的话轮转换预测。与基线模型相比，该模型表现更优，准确率达87.3%，宏F1分数为83.0%。\n随后我们开展了人类用户研究，在虚拟化身参与游戏《Dont Starve Together》（饥荒联机版）的交互场景中，对该话轮转换深度学习模型进行实证评估：将无话轮转换预测的控制组（n=20）与部署了我们模型的实验组（n=40）进行对比。由于已知话轮转换线索存在文化差异，两组均包含英语使用者和韩语使用者。我们随后分析了交互质量，考察了话语数量、打断频率以及参与者对虚拟化身的感知等维度。\n用户研究结果表明，我们的多模态话轮转换模型不仅提升了人机智能体对话的流畅度与自然度，还能在不显著改变对话频率的前提下维持平衡的会话动态。本研究深入揭示了话轮转换能力对用户感知与交互质量的影响，凸显了打造更具上下文适应性与响应性的对话智能体的潜力。
英文摘要：
This study investigates multimodal turn-taking prediction within human-agent interactions (HAI), particularly focusing on cooperative gaming environments. It comprises both model development and subsequent user study, aiming to refine our understanding and improve conversational dynamics in spoken dialogue systems (SDSs). For the modeling phase, we introduce a novel transformer-based deep learning (DL) model that simultaneously integrates multiple modalities - text, vision, audio, and contextual in-game data to predict turn-taking events in real-time. Our model employs a Crossmodal Transformer architecture to effectively fuse information from these diverse modalities, enabling more comprehensive turn-taking predictions. The model demonstrates superior performance compared to baseline models, achieving 87.3% accuracy and 83.0% macro F1 score. A human user study was then conducted to empirically evaluate the turn-taking DL model in an interactive scenario with a virtual avatar while playing the game "Dont Starve Together", comparing a control condition without turn-taking prediction (n=20) to an experimental condition with our model deployed (n=40). Both conditions included a mix of English and Korean speakers, since turn-taking cues are known to vary by culture. We then analyzed the interaction quality, examining aspects such as utterance counts, interruption frequency, and participant perceptions of the avatar. Results from the user study suggest that our multimodal turn-taking model not only enhances the fluidity and naturalness of human-agent conversations, but also maintains a balanced conversational dynamic without significantly altering dialogue frequency. The study provides in-depth insights into the influence of turn-taking abilities on user perceptions and interaction quality, underscoring the potential for more contextually adaptive and responsive conversational agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
