上下文感知的情感自适应语音助手：用于共情人类-智能体交互的多模态框架
Context-Aware Emotionally Adaptive Voice Assistants: A Multimodal Framework for Empathetic Human-Agent Interaction
- Vellore Institute of Technology, Amaravati（阿马拉瓦蒂维洛尔理工学院）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
针对语音助手打断的侵入性问题，提出多模态框架EmpathicVA，融合生理与语音信号及强化学习策略，实现情感自适应打断，显著提升满意度并减少压力。
中文摘要 AI 辅助
语音助手的打断往往具有侵入性，因为现有系统在决定何时以及如何打断用户时，未能考虑用户的情感状态、认知负荷和情境上下文。本文介绍了EmpathicVA，一个集成了生理感知、语音情感分析、情境建模和强化学习打断策略的闭环框架。该框架采用分层融合模型，整合心率活动（HRA）、皮电活动（EDA）、呼吸、声学韵律特征、语言嵌入和情境线索，并计算五种情感状态的概率。一个双深度Q网络（Double Deep Q-Network）根据这些概率、上下文和交互历史，选择立即响应、短暂或延长延迟、共情响应或静默模式。该多模态模型在留出测试集上取得了92.3%的准确率和0.922的宏平均F1分数，比准确率最高的单模态模型高出6.0个百分点。一项为期六周、包含48名参与者的受试者内现场研究，与基线和仅上下文感知的助手相比，在满意度、信任度和打断时机恰当性方面均有相应提升，同时与打断相关的压力事件大幅减少。结果表明，在语音交互中，除了响应措辞外，情感感知的时机把握和克制（弃权不执行）均至关重要。
英文摘要
Voice-assistant interruptions tend to be intrusive because existing systems fail to consider the affective state, cognitive load and situational context of the user when deciding when and how to interrupt.Voice-assistant interruptions tend to be intrusive, since existing systems do not consider the affective state, cognitive load or situational context of the user when determining when and how to interrupt. In this paper, EmpathicVA, a closed-loop framework integrating physiological sensing, vocal-affect analysis, contextual modeling and reinforcementlearning interruption policy, is introduced. A hierarchical fusion model involves integrating HRA, EDA, respiration, acousticprosodic features, linguistic embeddings, and contextual cues and computing the probabilities of five affective states. A Double Deep Q-Network selects immediate response, brief or extended delay, empathetic response, or silent mode based on these probabilities, context and interaction history. The multimodal model obtained an accuracy of 92.3% and an F1-score of 0.922 at the macro level on a held-out test set, outperforming the highest accuracy unimodal model by 6.0 percentage points. Comparing the six-week within-subject field study with 48 participants with a baseline and context-only assistants, there was a corresponding increase in satisfaction, trust, and appropriateness of timing, as well as a large reduction in interruption-related stress episodes. The results suggest that affect-aware timing and restraint are both important in voice interaction in addition to the response wording.
