Real-world application requires affect perception models to be sensitive to individual differences in expression. As each user is different and expresses differently, these models need to personalise towards each individual to adequately capture their expressions and thus, model their affective state. Despite high performance on benchmarks, current approaches fall short in such adaptation. In this work, we propose the use of Continual Learning (CL) for affective computing as a paradigm for developing personalised affect perception.
Bogazici University（博阿齐奇大学）；University of Cambridge（剑桥大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
As affective robots become integral in human life, these agents must be able to fairly evaluate human affective expressions without discriminating against specific demographic groups. Identifying bias in Machine Learning (ML) systems as a critical problem, different approaches have been proposed to mitigate such biases in the models both at data and algorithmic levels. In this work, we propose Continual Learning (CL) as an effective strategy to enhance fairness in Facial Expression Recognition (FER) systems, guarding against biases arising from imbalances in data distributions. We compare diff…
A Chinese Multi-label Affective Computing Dataset Based on Social Media Network Users
Beijing Institute of Technology（北京理工大学）；Institute of Scientific and Technical Research on Archives（档案科学技术研究所）；China Electronics Engineering Design Institute Co., Ltd.（中国电子工程设计院有限公司）
In recent years, the use of bio-sensing signals such as electroencephalogram (EEG), electrocardiogram (ECG), etc. have garnered interest towards applications in affective computing. The parallel trend of deep-learning has led to a huge leap in performance towards solving various vision-based research problems such as object detection. Yet, these advances in deep-learning have not adequately translated into bio-sensing research. This work applies novel deep-learning-based methods to various bio-sensing and video data of four publicly available multi-modal emotion datasets. For each dataset, we …
In recent years, transformer architecture has been a dominating paradigm in many applications, including affective computing. In this report, we propose our transformer-based model to handle Emotion Classification Task in the 5th Affective Behavior Analysis In-the-wild Competition. By leveraging the attentive model and the synthetic dataset, we attain a score of 0.4775 on the validation set of Aff-Wild2, the dataset provided by the organizer.
Early detection of psychological distress is key to effective treatment. Automatic detection of distress, such as depression, is an active area of research. Current approaches utilise vocal, facial, and bodily modalities. Of these, the bodily modality is the least investigated, partially due to the difficulty in extracting bodily representations from videos, and partially due to the lack of viable datasets. Existing body modality approaches use automatic categorization of expressions to represent body language as a series of specific expressions, much like words within natural language. In thi…
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen Technology University（深圳技术大学）；Tongji University（同济大学）；Huawei（华为）
Continual Learning at the Edge: An Agnostic IIoT Architecture
atlanTTic - I\&C Lab - Universidade de Vigo, Escola de Enxeñaría de Telecomunicación Campus Univesitario, Vigo 36310, Spain（atlanTTic - 信息与通信实验室 - 维戈大学，电信工程学院，维戈）
Federated Learning has been introduced as a new machine learning paradigm enhancing the use of local devices. At a server level, FL regularly aggregates models learned locally on distributed clients to obtain a more general model. Current solutions rely on the availability of large amounts of stored data at the client side in order to fine-tune the models sent by the server. Such setting is not realistic in mobile pervasive computing where data storage must be kept low and data characteristic can change dramatically. To account for this variability, a solution is to use the data regularly coll…
National University of Singapore（新加坡国立大学）；A*STAR Artificial Intelligence Initiative, Agency for Science, Technology and Research（新加坡科技研究局人工智能计划）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Oregon（俄勒冈大学）
Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, …
In neural networks, continual learning results in gradient interference among sequential tasks, leading to catastrophic forgetting of old tasks while learning new ones. This issue is addressed in recent methods by storing the important gradient spaces for old tasks and updating the model orthogonally during new tasks. However, such restrictive orthogonal gradient updates hamper the learning capability of the new tasks resulting in sub-optimal performance. To improve new learning while minimizing forgetting, in this paper we propose a Scaled Gradient Projection (SGP) method, where we combine th…
Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications
Electrical and Communication Engineering Department, United Arab Emirates University (UAEU), Al Ain, United Arab Emirates（阿联酋大学电子与通信工程系）；eBRAIN Lab, New York University (NYU) Abu Dhabi, United Arab Emirates（纽约大学阿布扎赫实验室）；National University of Sciences and Technology (NUST), Pakistan（巴基斯坦国立科学与技术大学）
Continual learning aims to alleviate catastrophic forgetting when handling consecutive tasks under non-stationary distributions. Gradient-based meta-learning algorithms have shown the capability to implicitly solve the transfer-interference trade-off problem between different examples. However, they still suffer from the catastrophic forgetting problem in the setting of continual learning, since the past data of previous tasks are no longer available. In this work, we propose a novel efficient meta-learning algorithm for solving the online continual learning problem, where the regularization t…
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Public Health, Peking University（北京大学公共卫生学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Peng Cheng Laboratory（鹏城实验室）
Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Warsaw University of Technology（华沙技术大学）；Computer Vision Center, Barcelona（巴塞罗那计算机视觉中心）；IDEAS Research Institute（IDEAS研究机构）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Xi’an Jiaotong University（西安交通大学）；Nankai University（南开大学）
Among the main features of biological intelligence are energy efficiency, capacity for continual adaptation, and risk management via uncertainty quantification. Neuromorphic engineering has been thus far mostly driven by the goal of implementing energy-efficient machines that take inspiration from the time-based computing paradigm of biological brains. In this paper, we take steps towards the design of neuromorphic systems that are capable of adaptation to changing learning tasks, while producing well-calibrated uncertainty quantification estimates. To this end, we derive online learning rules…
Thermal imaging-based physiological and affective computing is an emerging research area enabling technologies to monitor our bodily functions and understand psychological and affective needs in a contactless manner. However, up to recently, research has been mainly carried out in very controlled lab settings. As small size and even low-cost versions of thermal video cameras have started to appear on the market, mobile thermal imaging is opening its door to ubiquitous and real-world applications. Here we review the literature on the use of thermal imaging to track changes in physiological cues…
Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction
Social Machines & Robotics (SMART) Lab, and the Center of AI & Robotics (CAIR), New York University, Abu Dhabi（社会机器与机器人（SMART）实验室，以及人工智能与机器人中心（CAIR），纽约大学阿布扎克分校）；SMART Lab, New York University, Abu Dhabi（社会机器与机器人实验室，纽约大学阿布扎克分校）
University of California, Berkeley（加州大学伯克利分校）；University of Science and Technology of China（中国科学技术大学）；University of Southern California（南加州大学）；IBM Research AI（IBM人工智能研究院）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., image, music, and video), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. W…
In recent years, Affective Computing and its applications have become a fast-growing research topic. Furthermore, the rise of Deep Learning has introduced significant improvements in the emotion recognition system compared to classical methods. In this work, we propose a multi-modal emotion recognition model based on deep learning techniques using the combination of peripheral physiological signals and facial expressions. Moreover, we present an improvement to proposed models by introducing latent features extracted from our internal Bio Auto-Encoder (BAE). Both models are trained and evaluate…
Face based affective computing consists in detecting emotions from face images. It is useful to unlock better automatic comprehension of human behaviours and could pave the way toward improved human-machines interactions. However it comes with the challenging task of designing a computational representation of emotions. So far, emotions have been represented either continuously in the 2D Valence/Arousal space or in a discrete manner with Ekman's 7 basic emotions. Alternatively, Ekman's Facial Action Unit (AU) system have also been used to caracterize emotions using a codebook of unitary muscul…
GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing
The Hong Kong University of Science & Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science & Technology（香港科技大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Zhejiang University（浙江大学）；Great Bay University（大湾区大学）；Hangzhou Research Institute, Beihang University（北京航空航天大学杭州研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Institute of Digital Games University of Malta（马耳他大学数字游戏研究所）
This paper introduces a novel method for generating artistic images that express particular affective states. Leveraging state-of-the-art deep learning methods for visual generation (through generative adversarial networks), semantic models from OpenAI, and the annotated dataset of the visual art encyclopedia WikiArt, our AffectGAN model is able to generate images based on specific or broad semantic prompts and intended affective outcomes. A small dataset of 32 images generated by AffectGAN is annotated by 50 participants in terms of the particular emotion they elicit, as well as their quality…
Academia Sinica（中央研究院）；National Tsing Hua University（台湾清华大学）
Datasets with induced emotion labels are scarce but of utmost importance for many NLP tasks. We present a new, automated method for collecting texts along with their induced reaction labels. The method exploits the online use of reaction GIFs, which capture complex affective states. We show how to augment the data with induced emotion and induced sentiment labels. We use our method to create and publish ReactionGIF, a first-of-its-kind affective dataset of 30K tweets. We provide baselines for three new tasks, including induced sentiment prediction and multilabel classification of induced emoti…
This note gives a preliminary account of the transcoding or rechanneling problem between different stimuli as it is of interest for the natural interaction or affective computing fields. By the consideration of a simple example, namely the color response of an affective lamp to a sensed facial expression, we frame the problem within an information- theoretic perspective. A full justification in terms of the Information Bottleneck principle promotes a latent affective space, hitherto surmised as an appealing and intuitive solution, as a suitable mediator between the different stimuli.
Wuhan University（武汉大学）；Huazhong University of Science and Technology（华中科技大学）
Affective computing has become a very important research area in human-machine interaction. However, affects are subjective, subtle, and uncertain. So, it is very difficult to obtain a large number of labeled training samples, compared with the number of possible features we could extract. Thus, dimensionality reduction is critical in affective computing. This paper presents our preliminary study on dimensionality reduction for affect classification. Five popular dimensionality reduction approaches are introduced and compared. Experiments on the DEAP dataset showed that no approach can univers…
Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying…
Beijing Seek Truth Data Technology Co.,Ltd.（北京求真数据科技有限公司）
The Affective Behavior Analysis in-the-wild (ABAW) 2022 Competition gives Affective Computing a large promotion. In this paper, we present our method of AU challenge in this Competition. We use improved IResnet100 as backbone. Then we train AU dataset in Aff-Wild2 on three pertained models pretrained by our private au and expression dataset, and Glint360K respectively. Finally, we ensemble the results of our models. We achieved F1 score (macro) 0.731 on AU validation set.
MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild
School of Computer Science, CUG University of Geosciences (Wuhan)（中国地质大学（武汉）计算机学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
Faculty of Mathematics and Computer Science, Adam Mickiewicz University（数学与计算机科学学院，亚当·密茨凯维奇大学）；Faculty of Computer Science and Information Technology, Universiti Malaya（计算机科学与信息技术学院，马来亚大学）；Department of Computer Science and Engineering, Korea University（计算机科学与工程系，韩国大学）；Department of Psychology, University of Texas at Austin（心理学系，德克萨斯大学奥斯汀分校）；Cognitive Neuroscience Center, Adam Mickiewicz University（认知神经科学中心，亚当·密茨凯维奇大学）
