A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories
Department of Electronics, Information, and Bioengineering, Polytechnic University of Milan（米兰理工学院电子、信息与生物工程系）；Multimedia Mining and Search Group, Institute of Computational Perception, Johannes Kepler University Linz（莱茵河畔林茨约翰·凯撒大学计算感知研究所多媒体挖掘与检索小组）；Department of Music Pedagogy, Nuremberg University of Music（纽伦堡音乐学院音乐教育系）；Human-centered AI Group, AI Lab, Linz Institute of Technology（林茨技术学院以人为本的人工智能小组）
Fabio Ciani, Harald Schweiger, Emilia Parada-Cabaleiro, Markus Schedl
Text encoders are known for their utility in natural language processing, as they are able to efficiently compress inputs into dense vectors while preserving semantics. These models have been applied to affective computing, in particular to help with solving sentiment analysis and emotion recognition tasks. Nevertheless, it remains unclear to what extent the latent representations produced by modern text encoders capture well-defined psychological theories of affect. In this work, we investigate the affective capabilities of twelve recently released text encoders by probing their generated embeddings as input features for solving regression and classification tasks across three established emotion frameworks, using both word- and sentence-level data. Additionally, we apply a semantic data-leakage prevention technique to improve robustness in word-level evaluations. Our main findings show that the latent manifolds of the latest instruction-aware open-weight encoders enclose an equal or even a larger amount of affective information in comparison with proprietary counterparts when evaluated at word level. In contrast, embeddings of task-tuned and proprietary encoders reach the highest scores on sentence-level affective classification. Furthermore, a qualitative analysis of latent representations and their encoded affective cues is provided.
ETH Zurich（苏黎世联邦理工学院）；National Institute of Informatics（国立信息学研究所）
Emotions widely affect human decision-making. This fact is taken into account by affective computing with the goal of tailoring decision support to the emotional states of individuals. However, the accurate recognition of emotions within narrative documents presents a challenging undertaking due to the complexity and ambiguity of language. Performance improvements can be achieved through deep learning; yet, as demonstrated in this paper, the specific nature of this task requires the customization of recurrent neural networks with regard to bidirectional processing, dropout layers as a means of…
Universidad Politécnica de Madrid（马德里理工大学）；Telefónica（西班牙电信公司）；IFEMA（马德里展览中心）；Universidad Oberta De Catalunya（加泰罗尼亚开放大学）
Text classification tasks have improved substantially during the last years by the usage of transformers. However, the majority of researches focus on prose texts, with poetry receiving less attention, specially for Spanish language. In this paper, we propose a semi-supervised learning approach for inferring 21 psychological categories evoked by a corpus of 4572 sonnets, along with 10 affective and lexico-semantic multiclass ones. The subset of poems used for training an evaluation includes 270 sonnets. With our approach, we achieve an AUC beyond 0.7 for 76% of the psychological categories, an…
Affective Multimodal Agents with Proactive Knowledge Grounding for Emotionally Aligned Marketing Dialogue
Hunan Police Academy, Department of Criminal Investigation（湖南警察学院犯罪侦查系）；Business College, California State University（加州州立大学长滩分校商学院）；Northwestern University（西北大学）；AI Agent Lab, Vokram Group（Vokram集团AI代理实验室）；Beijing University of Technology（北京理工大学）；Zheng Zhou University of Light Industry（郑州轻工业大学）
Indian Institute of Technology Kanpur (IITK)（印度理工学院坎普尔分校（IITK））
Human use language not just to convey information but also to express their inner feelings and mental states. In this work, we adapt the state-of-the-art language generation models to generate affective (emotional) text. We posit a model capable of generating affect-driven and topic-focused sentences without losing grammatical correctness as the affect intensity increases. We propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2. The model gives a user the flexibility to control the category and intensity of emotion as well as the to…
Artificial Intelligence Application Research Center, Huawei Technologies（华为技术有限公司人工智能应用研究中心）
In this paper, we describe a data enhancement method for developing Emily, an emotion-affective open-domain chatbot. The proposed method is based on explicitly modeling positively transitioned (PT) sentiment data from multi-turn dialogues. We construct a dialogue corpus with PT sentiment data and will release it for public use. By fine-tuning a pretrained dialogue model using the produced PT-enhanced dialogues, we are able to develop an emotion-affective open-domain chatbot exhibiting close-to-human performance in various emotion-affective metrics. We evaluate Emily against a few state-of-the-…
Artificial Intelligence Application Research Center, Huawei Technologies（华为技术有限公司人工智能应用研究中心）
In this paper, we describe approaches for developing Emily, an emotion-affective open-domain chatbot. Emily can perceive a user's negative emotion state and offer supports by positively converting the user's emotion states. This is done by finetuning a pretrained dialogue model upon data capturing dialogue contexts and desirable emotion states transition across turns. Emily can differentiate a general open-domain dialogue utterance with questions relating to personal information. By leveraging a question-answering approach based on knowledge graphs to handle personal information, Emily maintai…
EmoMind: Decoding Affective Captions from Human Brain fMRI
Department of Biomedical Engineering（生物医学工程系）；Vanderbilt University（范德比大学）；Research Institute of Electrical Communication（电气通信研究所）；Tohoku University（东北大学）；University of Florida（佛罗里达大学）
Telefónica（西班牙电信）；Universidad Nacional de Educación a Distancia (UNED)（国立远程教育大学）
Nowadays, there are many applications of text mining over corpora from different languages. However, most of them are based on texts in prose, lacking applications that work with poetry texts. An example of an application of text mining in poetry is the usage of features derived from their individual words in order to capture the lexical, sublexical and interlexical meaning, and infer the General Affective Meaning (GAM) of the text. However, even though this proposal has been proved as useful for poetry in some languages, there is a lack of studies for both Spanish poetry and for highly-struct…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））
Datasets that capture the connection between vision, language, and affection are limited, causing a lack of understanding of the emotional aspect of human intelligence. As a step in this direction, the ArtEmis dataset was recently introduced as a large-scale dataset of emotional reactions to images along with language explanations of these chosen emotions. We observed a significant emotional bias towards instance-rich emotions, making trained neural speakers less accurate in describing under-represented emotions. We show that collecting new data, in the same way, is not effective in mitigating…
A Chinese Multi-label Affective Computing Dataset Based on Social Media Network Users
Beijing Institute of Technology（北京理工大学）；Institute of Scientific and Technical Research on Archives（档案科学技术研究所）；China Electronics Engineering Design Institute Co., Ltd.（中国电子工程设计院有限公司）
University of Sheffield（谢菲尔德大学）；Utrecht University（乌得勒支大学）；Zhejiang Wanli University（浙江万里大学信息与智能工程学院）
Understanding speaker's feelings and producing appropriate responses with emotion connection is a key communicative skill for empathetic dialogue systems. In this paper, we propose a simple technique called Affective Decoding for empathetic response generation. Our method can effectively incorporate emotion signals during each decoding step, and can additionally be augmented with an auxiliary dual emotion encoder, which learns separate embeddings for the speaker and listener given the emotion base of the dialogue. Extensive empirical studies show that our models are perceived to be more empath…
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
Center for Language and Speech Processing (CLSP), Johns Hopkins University（约翰霍普金斯大学语言与语音处理中心）；Group on Language, Audio & Music (GLAM), Imperial College London（伦敦帝国理工学院语言、音频与音乐组）
Georgia Tech（佐治亚理工学院）；Facebook AI Research（脸书人工智能研究院）
We present Lemotif, an integrated natural language processing and image generation system that uses machine learning to (1) parse a text-based input journal entry describing the user's day for salient themes and emotions and (2) visualize the detected themes and emotions in creative and appealing image motifs. Synthesizing approaches from artificial intelligence and psychology, Lemotif acts as an affective visual journal, encouraging users to regularly write and reflect on their daily experiences through visual reinforcement. By making patterns in emotions and their sources more apparent, Lemo…
Academia Sinica（中央研究院）；Carnegie Mellon University（卡内基梅隆大学）
Instant messaging is one of the major channels of computer mediated communication. However, humans are known to be very limited in understanding others' emotions via text-based communication. Aiming on introducing emotion sensing technologies to instant messaging, we developed EmotionPush, a system that automatically detects the emotions of the messages end-users received on Facebook Messenger and provides colored cues on their smartphones accordingly. We conducted a deployment study with 20 participants during a time span of two weeks. In this paper, we revealed five challenges, along with ex…
Effectively recognising and applying emotions to interactions is a highly desirable trait for social robots. Implicitly understanding how subjects experience different kinds of actions and objects in the world is crucial for natural HRI interactions, with the possibility to perform positive actions and avoid negative actions. In this paper, we utilize the NICO robot's appearance and capabilities to give the NICO the ability to model a coherent affective association between a perceived auditory stimulus and a temporally asynchronous emotion expression. This is done by combining evaluations of e…
University of Southern California（南加州大学）；Carnegie Mellon University（卡内基梅隆大学）
Human verbal communication includes affective messages which are conveyed through use of emotionally colored words. There has been a lot of research in this direction but the problem of integrating state-of-the-art neural language models with affective information remains an area ripe for exploration. In this paper, we propose an extension to an LSTM (Long Short-Term Memory) language model for generating conversational text, conditioned on affect categories. Our proposed model, Affect-LM enables us to customize the degree of emotional content in generated sentences through an additional design…
University of Applied Sciences Potsdam（波茨坦应用科学大学）；Urban Futures Institute of University of Applied Sciences Potsdam（波茨坦应用科学大学城市未来研究所）
This paper presents Emosaic, a tool for visualizing the emotional tone of text documents, considering multiple dimensions of emotion and varying levels of semantic granularity. Emosaic is grounded in psychological research on the relationship between language, affect, and color perception. We capitalize on an established three-dimensional model of human emotion: valence (good, nice vs. bad, awful), arousal (calm, passive vs. exciting, active) and dominance (weak, controlled vs. strong, in control). Previously, multi-dimensional models of emotion have been used rarely in visualizations of textu…
Deep Learning-Based Feature Fusion for Emotion Analysis and Suicide Risk Differentiation in Chinese Psychological Support Hotlines
School of Software Engineering, Beijing University of Technology（北京工业大学软件学院）；Peking University Huilongguan Clinical Medical School（北京大学回龙湖临床医学院）；Sorbonne Université（索邦大学）；Institut du Cerveau – Paris Brain Institute - ICM（巴黎脑研究所（ICM））；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗集团）；Hôpital de la Pitié-Salpêtrière（皮蒂-萨尔皮特里医院）
Cheriton School of Computer Science, University of Waterloo（滑铁卢大学切里顿计算机科学学院）
Existing neural conversational models process natural language primarily on a lexico-syntactic level, thereby ignoring one of the most crucial components of human-to-human dialogue: its affective content. We take a step in this direction by proposing three novel ways to incorporate affective/emotional aspects into long short term memory (LSTM) encoder-decoder neural conversation models: (1) affective word embeddings, which are cognitively engineered, (2) affect-based objective functions that augment the standard cross-entropy loss, and (3) affectively diverse beam search for decoding. Experime…
GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations
Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）；Konrad Zuse School of Excellence in Reliable AI（康拉德·祖斯可靠人工智能卓越学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Stanford University（斯坦福大学）；Ecole Polytechnique, IP Paris（巴黎理工学院）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language. In contrast to most existing annotation datasets in computer vision, we focus on the affective experience triggered by visual artworks and ask the annotators to indicate the dominant emotion they feel for a given image and, crucially, to also provide a grounded verbal explanation for their emotion choice. As we demonstrate below, this leads to a rich set of signals …
