发表机构
Khoury College of Computer Science; Northeastern University（计算机科学学院（科里学院）; 东北大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对虚拟人类言语与非言语行为的不一致问题，提出分类法，探究LLM对情境适配的不匹配行为的选择能力，并通过人类受试者研究验证其效果。
AI 中文摘要
虚拟智能体的非言语行为生成系统通常以言语内容为输入，生成用于强调或说明言语内容的非言语行为。然而，人类的非言语行为不仅受言语内容影响，还受说话者角色、人际关系、社会情境以及交互双方的认知和情绪状态影响。因此，非言语通道可能会强化、弱化、限定甚至与言语通道矛盾，还可能揭示言语中隐藏或间接暗示的内部状态，包括即时交互中偶然出现的情绪“泄露”。为设计出具有真实人类行为的虚拟智能体，对言语与非言语行为间更丰富的关系进行建模十分重要，在需要精细社会解读的训练情境（如涉及虚拟患者的咨询模拟）中尤为关键。本文借鉴Ekman的言语-非言语关系框架，提出了言语与非言语行为间可能出现不匹配的类别分类法；随后研究了利用大语言模型（LLM）实现此类行为的不同方法，重点探究LLM能否从给定对话和社会交互情境中选择出符合情境的不匹配言语与非言语行为；最后通过人类受试者研究对生成的行为进行评估，判断当情境驱动的非言语行为具象化为虚拟人类时，是否会对观察者产生预期效果。
英文摘要
Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate the content of the verbal channel. However, human nonverbal behavior is shaped by more than the content of the speech. It is also influenced by speaker roles, interpersonal relationships, social context, and the cognitive and emotional states of the interactants. As a result, the nonverbal channel may reinforce, weaken, qualify, or even contradict the verbal channel. It may also reveal internal states that are hidden or only indirectly implied in speech, including emotional "leakage" that may be incidental to the immediate interaction. Modeling this richer relationship between verbal and nonverbal behavior is important for designing virtual agents that exhibit realistic, human-like behavior. It is especially critical in training contexts that require nuanced social interpretation, such as counseling simulations involving virtual patients. Drawing on Ekman's framework of verbal nonverbal relationships, we propose a taxonomy of categories in which mismatches between verbal and nonverbal behavior can occur. We then examine alternative approaches for realizing these behaviors using large language models, focusing on whether LLMs can select contextually appropriate mismatched verbal and nonverbal behaviors from a given dialogue and social interaction context. Finally, we evaluate the resulting behaviors in a human-subject study, assessing whether context-driven nonverbal behavior, when embodied in a virtual human, produces the intended effects on observers.
