多语言二元交互语料库 NoXi+J：迈向理解亚欧非语言文化特征及其对参与度的影响
Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement
AI总结：
本研究构建了多语言二元交互数据集 NoXi+J，通过提取多模态非语言特征并利用 LSTM 模型与 SHAP 分析，揭示了亚欧文化差异对参与度预测特征重要性的显著影响。
摘要：
非语言行为是理解对话动态以及交互中对话者之间产生的情感状态的核心挑战。尽管心理学研究表明非语言行为在不同文化间存在差异，但目前进行的计算分析有限，难以阐明这些差异并评估其对参与度识别的影响。为了更深入地理解广泛文化和语言圈中的参与度和非语言行为，本研究对非语言特征进行了多语言计算分析，并探讨了它们在参与度及参与度预测中的作用。为实现这一目标，我们首先扩展了 NoXi 数据集（该数据集包含居住在法国、德国和英国的参与者的交互数据），通过收集日语和中文的二元对话会话数据，构建了增强型数据集 NoXi+J。接着，我们通过各种模式识别技术和算法提取了多模态非语言特征，包括语音声学、面部表情、回应反馈和手势。然后，我们对倾听行为和回应反馈模式进行了统计分析，以识别每种语言中具有文化依赖性和独立性的特征，以及多种语言间的共同特征。这些特征也与对话者表现出的参与度进行了相关性分析。最后，我们分析了在五种语言数据集上训练用于预测参与度的 LSTM 模型的输入特征中文化差异的影响。结合迁移学习的 SHAP 分析证实，语言集的输入特征重要性与所分析的显著文化特征之间存在显著的相关性。
英文摘要：
Non-verbal behavior is a central challenge in understanding the dynamics of a conversation and the affective states between interlocutors arising from the interaction. Although psychological research has demonstrated that non-verbal behaviors vary across cultures, limited computational analysis has been conducted to clarify these differences and assess their impact on engagement recognition. To gain a greater understanding of engagement and non-verbal behaviors among a wide range of cultures and language spheres, in this study we conduct a multilingual computational analysis of non-verbal features and investigate their role in engagement and engagement prediction. To achieve this goal, we first expanded the NoXi dataset, which contains interaction data from participants living in France, Germany, and the United Kingdom, by collecting session data of dyadic conversations in Japanese and Chinese, resulting in the enhanced dataset NoXi+J. Next, we extracted multimodal non-verbal features, including speech acoustics, facial expressions, backchanneling and gestures, via various pattern recognition techniques and algorithms. Then, we conducted a statistical analysis of listening behaviors and backchannel patterns to identify culturally dependent and independent features in each language and common features among multiple languages. These features were also correlated with the engagement shown by the interlocutors. Finally, we analyzed the influence of cultural differences in the input features of LSTM models trained to predict engagement for five language datasets. A SHAP analysis combined with transfer learning confirmed a considerable correlation between the importance of input features for a language set and the significant cultural characteristics analyzed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
