MemEmo: Evaluating Emotion in Memory Systems of Agents
School of Information, Renmin University of China（中国人民大学信息学院）；China Telecom Research Institute（中国电信研究院）；MemTensor (Shanghai) Technology（MemTensor（上海）科技）
Memory systems address the challenge of context loss in Large Language Model during prolonged interactions. However, compared to human cognition, the efficacy of these systems in processing emotion-related information remains inconclusive. To address this gap, we propose an emotion-enhanced memory evaluation benchmark to assess the performance of mainstream and state-of-the-art memory systems in handling affective information. We developed the \textbf{H}uman-\textbf{L}ike \textbf{M}emory \textbf{E}motion (\textbf{HLME}) dataset, which evaluates memory systems across three dimensions: emotional information extraction, emotional memory updating, and emotional memory question answering. Experimental results indicate that none of the evaluated systems achieve robust performance across all three tasks. Our findings provide an objective perspective on the current deficiencies of memory systems in processing emotional memories and suggest a new trajectory for future research and system optimization.
EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models
Fudan University（复旦大学）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Beijing Yuandian Technology（北京原点科技）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding
Instituto Politécnico Nacional (IPN)（墨西哥国立理工学院）；Wollo University（沃洛大学）；Saarland University（萨尔大学）；Bahir Dar University（巴赫达尔大学）；University of Hamburg（汉堡大学）
Current approaches to speech emotion recognition focus on speech features that can capture the emotional content of a speech signal. Mel Frequency Cepstral Coefficients (MFCCs) are one of the most commonly used representations for audio speech recognition and classification. This paper proposes Gammatone Frequency Cepstral Coefficients (GFCCs) as a potentially better representation of speech signals for emotion recognition. The effectiveness of MFCC and GFCC representations are compared and evaluated over emotion and intensity classification tasks with fully connected and recurrent neural netw…
When Seekers Are Hard to Help: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning（湖北人工智能与智能学习省级重点实验室）；National Language Resources Monitoring and Research Center for Network Media（网络媒体语言资源监测与研究中心）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）；Faculty of Artificial Intelligence in Education, Central China Normal University（华中师范大学教育人工智能学院）；School of Chinese Language and Literature, Central China Normal University（华中师范大学中文语言文学学院）
EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
Lappeenranta-Lahti University of Technology LUT（拉佩兰塔-拉赫蒂技术大学）；University of Oulu（奥卢大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Brno University of Technology（布拉格技术大学）
Emotion lexicons describe the affective meaning of words and thus constitute a centerpiece for advanced sentiment and emotion analysis. Yet, manually curated lexicons are only available for a handful of languages, leaving most languages of the world without such a precious resource for downstream applications. Even worse, their coverage is often limited both in terms of the lexical units they contain and the emotional variables they feature. In order to break this bottleneck, we here introduce a methodology for creating almost arbitrarily large emotion lexicons for any target language. Our app…
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Tianjin University（天津大学）；Fuzhou University（福州大学）；Shanghai Jiaotong University（上海交通大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
ESC-Eval: Evaluating Emotion Support Conversations in Large Language Models
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Information, Renmin University of China（中国人民大学信息学院）
Privacy preservation is a crucial component of any real-world application. But, in applications relying on machine learning backends, privacy is challenging because models often capture more than what the model was initially trained for, resulting in the potential leakage of sensitive information. In this paper, we propose an automatic and quantifiable metric that allows us to evaluate humans' perception of a model's ability to preserve privacy with respect to sensitive variables. In this paper, we focus on saliency-based explanations, explanations that highlight regions of the input text, to …
Customer service is a setting that calls for empathy in live human agent responses. Recent advances have demonstrated how open-domain chatbots can be trained to demonstrate empathy when responding to live human utterances. We show that a blended skills chatbot model that responds to customer queries is more likely to resemble actual human agent response if it is trained to recognize emotion and exhibit appropriate empathy, than a model without such training. For our analysis, we leverage a Twitter customer service dataset containing several million customer<->agent dialog examples in customer …
Decoding Neural Emotion Patterns through Large Language Model Embeddings
College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；College of Health Care Sciences, James Cook University（健康护理科学学院，詹姆斯库克大学）；College of Public Health, Medical, and Vet Sciences, James Cook University（公共健康、医学与兽医学学院，詹姆斯库克大学）
The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youth
Baruch Ivcher School of Psychology, Reichman University（赖希曼大学巴鲁克·伊夫切尔心理学院）；The Education Department, Ariel University（阿里尔大学教育系）；Rambam Medical Center（兰巴姆医疗中心）；Faculty of Data and Decision Sciences, Technion, IIT（以色列理工学院数据与决策科学学院）
National University of Singapore（新加坡国立大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Southern Queensland（南昆士兰大学）；Imperial College London（伦敦帝国理工学院）；University of Augsburg（奥格斯堡大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Bremen（不来梅大学）
Emotional speech synthesis aims to synthesize human voices with various emotional effects. The current studies are mostly focused on imitating an averaged style belonging to a specific emotion type. In this paper, we seek to generate speech with a mixture of emotions at run-time. We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. During the training, the framework does not only explicitly characterize emotion styles, but also ex…
Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm
Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）；University of Belgrade（贝尔格莱德大学）；Institute for Philosophy and Social Theory（哲学与社会理论研究所）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)（莱茵兰-普法尔茨州 Kaiserslautern-Landau 应用科技大学）；German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Vilnius Gediminas Technical University（维尔纽斯格迪米纳斯技术大学）；Drexel University（德雷塞尔大学）；University of Montenegro（黑山大学）；Faculty of Dramatic Arts（戏剧艺术学院）；University of Kragujevac（克拉古耶瓦茨大学）；Faculty of Engineering（工程学院）
EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence（广东人工智能与数字经济实验室）；Auckland University of Technology（奥克兰理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；SKL-IOTSC, CIS, University of Macau（澳门科学技术大学SKL-IOTSC、CIS）
