We introduce the concept of "empathic grounding" in conversational agents as an extension of Clark's conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the speaker's affective state. Empathic grounding is generally required whenever the speaker's emotions are foregrounded and can make the grounding process more efficient and reliable by communicating both propositional and affective understanding. Both speaker expressions of affect and listener empathic grounding can be multimodal, including facial expressions and other nonverbal displays. Thus, models of empathic grounding for embodied agents should be multimodal to facilitate natural and efficient communication. We describe a multimodal model that takes as input user speech and facial expression to generate multimodal grounding moves for a listening agent using a large language model. We also describe a testbed to evaluate approaches to empathic grounding, in which a humanoid robot interviews a user about a past episode of pain and then has the user rate their perception of the robot's empathy. We compare our proposed model to one that only generates non-affective grounding cues in a between-subjects experiment. Findings demonstrate that empathic grounding increases user perceptions of empathy, understanding, emotional intelligence, and trust. Our work highlights the role of emotion awareness and multimodality in generating appropriate grounding moves for conversational agents.
Practicing with Language Models Cultivates Human Empathic Communication
Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Northwestern Institute for Complex Systems, Northwestern University（西北大学复杂系统研究所）；Ryan Institute on Complexity, Northwestern University（西北大学复杂性研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Department of Communication Studies, Northwestern University（西北大学传播学系）；Department of Computer Science, Northwestern University（西北大学计算机科学系）
When Large Language Models are Reliable for Judging Empathic Communication
Kellogg School of Management（凯洛格管理学院）；Northwestern University（西北大学）；Northwestern Institute for Complex Systems（西北复杂系统研究所）；Stanford University（斯坦福大学）；Pennsylvania State University（宾夕法尼亚州立大学）
There is a growing recognition that artists use valuable ways to understand and work with cognitive and perceptual mechanisms to convey desired experiences and narrative in their created artworks (DiPaola et al., 2010; Zeki, 2001). This paper documents our attempt to computationally model the creative process of a portrait painter, who relies on understanding human traits (i.e., personality and emotions) to inform their art. Our system includes an empathic conversational interaction component to capture the dominant personality category of the user and a generative AI Portraiture system that u…
Conversational chatbots are Artificial Intelligence (AI)-powered applications that assist users with various tasks by responding in natural language and are prevalent across different industries. Most of the chatbots that we encounter on websites and digital assistants such as Alexa, Siri does not express empathy towards the user, and their ability to empathise remains immature. Lack of empathy towards the user is not critical for a transactional or interactive chatbot, but the bots designed to support mental healthcare patients need to understand the emotional state of the user and tailor the…
chatClimate: Grounding Conversational AI in Climate Science
University of Zurich（苏黎世大学）；Eawag（瑞士水科学与技术研究所）；ETH Zurich（苏黎世联邦理工学院）；FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；University of Oxford（牛津大学）；Council on Economic Policies（经济政策委员会）；Swiss Finance Institute (SFI)（瑞士金融学院）
Humans usually have conversations by making use of prior knowledge about a topic and background information of the people whom they are talking to. However, existing conversational agents and datasets do not consider such comprehensive information, and thus they have a limitation in generating the utterances where the knowledge and persona are fused properly. To address this issue, we introduce a call For Customized conversation (FoCus) dataset where the customized answers are built with the user's persona and Wikipedia knowledge. To evaluate the abilities to make informative and customized ut…
Hear You in Silence: Designing for Active Listening in Human Interaction with Conversational Agents Using Context-Aware Pacing
The University of Hong Kong Hong Kong, SAR China Columbia University New York United States；Renmin University of China Beijing China；City University of Hong Kong\ for Narrative Spaces Hong Kong, SAR China Guangdong University of Technology Guangzhou China；City University of Hong Kong\ for Narrative Spaces Hong Kong, SAR China；The University of Hong Kong；Columbia University；Renmin University of China；City University of Hong Kong\ for Narrative Spaces；Guangdong University of Technology
Korea University（高丽大学）；NCSOFT（NCSOFT公司）；Naver Corporation（Naver公司）
To build a conversational agent that interacts fluently with humans, previous studies blend knowledge or personal profile into the pre-trained language model. However, the model that considers knowledge and persona at the same time is still limited, leading to hallucination and a passive way of using personas. We propose an effective dialogue agent that grounds external knowledge and persona simultaneously. The agent selects the proper knowledge and persona to use for generating the answers with our candidate scoring implemented with a poly-encoder. Then, our model generates the utterance with…
University of Washington（华盛顿大学）；Stanford University（斯坦福大学）
Advances in artificial intelligence (AI) are enabling systems that augment and collaborate with humans to perform simple, mechanistic tasks like scheduling meetings and grammar-checking text. However, such Human-AI collaboration poses challenges for more complex, creative tasks, such as carrying out empathic conversations, due to difficulties of AI systems in understanding complex human emotions and the open-ended nature of these tasks. Here, we focus on peer-to-peer mental health support, a setting in which empathy is critical for success, and examine how AI can collaborate with humans to fac…
Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog generation without relying on extra knowledge annotation. Specifically, we use a pre-trained language model to extract the most uncertain tok…
Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts
The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Singapore Management University（新加坡管理学院）；The Hong Kong Polytechnic University（香港理工大学）
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Talk2Escape: Conversational Grounding for Vision-and-Language Navigation
Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）；Zhejiang Wanli University（浙江万里学院）；Fudan University（复旦大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs
Amsterdam UMC, Department of Medical Informatics（阿姆斯特丹大学医学中心，医学信息学系）；Center for Information and Language Processing, LMU Munich（信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML), Munich（慕尼黑机器学习中心（MCML））；Institute for Logic, Language and Computation (ILLC), University of Amsterdam（逻辑、语言与计算研究所（ILLC），阿姆斯特丹大学）
