In recent years, Large Language Models (LLMs) have become increasingly more powerful in their ability to complete complex tasks. One such task in which LLMs are often employed is scoring, i.e., assigning a numerical value from a certain scale to a subject. In this paper, we strive to understand how LLMs score, specifically in the context of empathy scoring. We develop a novel and comprehensive framework for investigating how effective LLMs are at measuring and scoring empathy of responses in dialogues, and what methods can be employed to deepen our understanding of LLM scoring. Our strategy is to approximate the performance of state-of-the-art and fine-tuned LLMs with explicit and explainable features. We train classifiers using various features of dialogues including embeddings, the Motivational Interviewing Treatment Integrity (MITI) Code, a set of explicit subfactors of empathy as proposed by LLMs, and a combination of the MITI Code and the explicit subfactors. Our results show that when only using embeddings, it is possible to achieve performance close to that of generic LLMs, and when utilizing the MITI Code and explicit subfactors scored by an LLM, the trained classifiers can closely match the performance of fine-tuned LLMs. We employ feature selection methods to derive the most crucial features in the process of empathy scoring. Our work provides a new perspective toward understanding LLM empathy scoring and helps the LLM community explore the potential of LLM scoring in social science studies.
Generative adversarial imitation learning (GAIL) is a model-free algorithm that has been shown to provide strong results in imitating complex behaviors in high-dimensional environments. In this paper, we utilize the GAIL model for text generation to develop empathy-based context-aware conversational AI. Our model uses an expert trajectory of empathetic prompt-response dialogues which can accurately exhibit the correct empathetic emotion when generating a response. The Generator of the GAIL model uses the GPT-2 sequential pre-trained language model trained on 117 million parameters from 40 GB o…
Empathy by Design: Aligning Large Language Models for Healthcare Dialogue
Department of Public Health Sciences, The University of Texas at El Paso, USA（公共卫生科学系，德克萨斯理工大学埃尔帕索分校）；Department of Nursing, The University of Texas at El Paso, USA（护理系，德克萨斯理工大学埃尔帕索分校）；Department of Rehabilitation Sciences, The University of Texas at El Paso, USA（康复科学系，德克萨斯理工大学埃尔帕索分校）
EmplifAI: a Fine-grained Dataset for Japanese Empathetic Medical Dialogues in 28 Emotion Labels
Kyoto Institute of Technology, Japan（京都技术大学）；National Institute of Information and Communications Technology (NICT), Japan（信息通信技术国家研究所）；Kyoto University, Japan（京都大学）；Nara Institute of Science and Technology, Japan（奈良科学技术大学）
Efficient-Empathy: Towards Efficient and Effective Selection of Empathy Data
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室）；SKL-IOTSC, CIS, University of Macau（澳门科学馆物联网与智慧城市研究中心）；Shandong University（山东大学）；Auckland University of Technology（技术大学（奥克兰））；University of Electronic Science and Technology of China（电子科技大学）
APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation
Sun Yat-Sen University（中山大学）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Sibyl: Empowering Empathetic Dialogue Generation in Large Language Models via Sensible and Visionary Commonsense Inference
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；WeChat AI, Tencent Inc（腾讯微信人工智能）；Meituan Group（美团集团）
PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation
Anhui Robot Technology Standard Innovation Base, School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院安徽机器人技术标准创新基地）
FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；ELLIS Institute Tübingen（ELLIS图宾根研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）
The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability
Department of Computer Science and Technology, Tsinghua University, China（清华大学计算机科学与技术系）；Institute for AI Industry Research (AIR), Tsinghua University, China（清华大学人工智能产业研究院）
Korea University（高丽大学）；NCSOFT（NCSOFT公司）；Naver Corporation（Naver公司）
To build a conversational agent that interacts fluently with humans, previous studies blend knowledge or personal profile into the pre-trained language model. However, the model that considers knowledge and persona at the same time is still limited, leading to hallucination and a passive way of using personas. We propose an effective dialogue agent that grounds external knowledge and persona simultaneously. The agent selects the proper knowledge and persona to use for generating the answers with our candidate scoring implemented with a poly-encoder. Then, our model generates the utterance with…
WeChat AI, Tencent Inc（腾讯公司微信人工智能实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）
We participate in the DSTC9 Interactive Dialogue Evaluation Track (Gunasekara et al. 2020) sub-task 1 (Knowledge Grounded Dialogue) and sub-task 2 (Interactive Dialogue). In sub-task 1, we employ a pre-trained language model to generate topic-related responses and propose a response ensemble method for response selection. In sub-task2, we propose a novel Dialogue Planning Model (DPM) to capture conversation flow in the interaction with humans. We also design an integrated open-domain dialogue system containing pre-process, dialogue model, scoring model, and post-process, which can generate flu…
Practicing with Language Models Cultivates Human Empathic Communication
Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Northwestern Institute for Complex Systems, Northwestern University（西北大学复杂系统研究所）；Ryan Institute on Complexity, Northwestern University（西北大学复杂性研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Department of Communication Studies, Northwestern University（西北大学传播学系）；Department of Computer Science, Northwestern University（西北大学计算机科学系）
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
MotivGraph-SoIQ: Integrating Motivational Knowledge Graphs and Socratic Dialogue for Enhanced LLM Ideation
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems（认知与决策智能复杂系统重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Hunan Provincial Key Laboratory of Philosophy and Social Sciences of Artificial Intelligence and Precision International, Hunan Normal University（湖南省人工智能与精准哲学社会科学省级重点实验室，湖南师范大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Northeastern University（东北大学）；Swinburne University of Technology（斯威本科技大学）
Empathy is an important characteristic to be considered when building a more intelligent and humanized dialogue agent. However, existing methods did not fully comprehend empathy as a complex process involving three aspects: cognition, affection and behavior. In this paper, we propose CAB, a novel framework that takes a comprehensive perspective of cognition, affection and behavior to generate empathetic responses. For cognition, we build paths between critical keywords in the dialogue by leveraging external knowledge. This is because keywords in a dialogue are the core of sentences. Building t…
Dialogue topic segmentation is critical in several dialogue modeling problems. However, popular unsupervised approaches only exploit surface features in assessing topical coherence among utterances. In this work, we address this limitation by leveraging supervisory signals from the utterance-pair coherence scoring task. First, we present a simple yet effective strategy to generate a training corpus for utterance-pair coherence scoring. Then, we train a BERT-based neural utterance-pair coherence model with the obtained training corpus. Finally, such model is used to measure the topical relevanc…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Pattern Recognition Center, WeChat AI, Tencent Inc（腾讯公司微信人工智能模式识别中心）
Researches on dialogue empathy aim to endow an agent with the capacity of accurate understanding and proper responding for emotions. Existing models for empathetic dialogue generation focus on the emotion flow in one direction, that is, from the context to response. We argue that conducting an empathetic conversation is a bidirectional process, where empathy occurs when the emotions of two interlocutors could converge on the same point, i.e., reaching an emotion consensus. Besides, we also find that the empathetic dialogue corpus is extremely limited, which further restricts the model performa…
Previous research on empathetic dialogue systems has mostly focused on generating responses given certain emotions. However, being empathetic not only requires the ability of generating emotional responses, but more importantly, requires the understanding of user emotions and replying appropriately. In this paper, we propose a novel end-to-end approach for modeling empathy in dialogue systems: Mixture of Empathetic Listeners (MoEL). Our model first captures the user emotions and outputs an emotion distribution. Based on this, MoEL will softly combine the output states of the appropriate Listen…
