Early detection of psychological distress is key to effective treatment. Automatic detection of distress, such as depression, is an active area of research. Current approaches utilise vocal, facial, and bodily modalities. Of these, the bodily modality is the least investigated, partially due to the difficulty in extracting bodily representations from videos, and partially due to the lack of viable datasets. Existing body modality approaches use automatic categorization of expressions to represent body language as a series of specific expressions, much like words within natural language. In this dissertation I present a new type of feature, within the body modality, that represents meta information of gestures, such as speed, and use it to predict a non-clinical depression label. This differs to existing work by representing overall behaviour as a small set of aggregated meta features derived from a person's movement. In my method I extract pose estimation from videos, detect gestures within body parts, extract meta information from individual gestures, and finally aggregate these features to generate a small feature vector for use in prediction tasks. I introduce a new dataset of 65 video recordings of interviews with self-evaluated distress, personality, and demographic labels. This dataset enables the development of features utilising the whole body in distress detection tasks. I evaluate my newly introduced meta-features for predicting depression, anxiety, perceived stress, somatic stress, five standard personality measures, and gender. A linear regression based classifier using these features achieves a 82.70% F1 score for predicting depression within my novel dataset.
SGED: A Benchmark dataset for Performance Evaluation of Spiking Gesture Emotion Recognition
Shandong Massive Information Technology Research Institute（山东海量信息技术研究院）；State Key Laboratory of High-end Server & Storage Technology（高端服务器与存储技术国家重点实验室）；Inspur (Beijing) Electronic Information Industry Co., Ltd.（浪潮（北京）电子信息产业有限公司）；Inspur Electronic Information Industry Co., Ltd.（浪潮电子信息产业股份有限公司）；Intel Labs China（英特尔中国研究院）
Although psychological research indicates that bodily expressions convey important affective information, to date research in emotion recognition focused mainly on facial expression or voice analysis. In this paper we propose an approach to realtime automatic emotion recognition from body movements. A set of postural, kinematic, and geometrical features are extracted from sequences 3D skeletons and fed to a multi-class SVM classifier. The proposed method has been assessed on data acquired through two different systems: a professionalgrade optical motion capture system, and Microsoft Kinect. Th…
Stanford University（斯坦福大学）；Ecole Polytechnique, IP Paris（巴黎理工学院）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language. In contrast to most existing annotation datasets in computer vision, we focus on the affective experience triggered by visual artworks and ask the annotators to indicate the dominant emotion they feel for a given image and, crucially, to also provide a grounded verbal explanation for their emotion choice. As we demonstrate below, this leads to a rich set of signals …
Collaborative learning is an educational approach that enhances learning through shared goals and working together. Interaction and regulation are two essential factors related to the success of collaborative learning. Since the information from various modalities can reflect the quality of collaboration, a new multimodal dataset with cognitive and emotional triggers is introduced in this paper to explore how regulations affect interactions during the collaborative process. Specifically, a learning task with intentional interventions is designed and assigned to high school students aged 15 yea…
Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
Faculty of Mathematics and Computer Science, Adam Mickiewicz University（数学与计算机科学学院，亚当·密茨凯维奇大学）；Faculty of Computer Science and Information Technology, Universiti Malaya（计算机科学与信息技术学院，马来亚大学）；Department of Computer Science and Engineering, Korea University（计算机科学与工程系，韩国大学）；Department of Psychology, University of Texas at Austin（心理学系，德克萨斯大学奥斯汀分校）；Cognitive Neuroscience Center, Adam Mickiewicz University（认知神经科学中心，亚当·密茨凯维奇大学）
BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer
Guangdong Academy of Sciences（广东省科学院）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；Meta Reality Labs Research（Meta现实实验室）；National Institute of Informatics（国立信息学研究所）；Tohoku University（东北大学）
GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos
Department of Computer Science and Engineering, Indian Institute of Technology Roorkee（计算机科学与工程系，印度理工学院Roorkee分校）；Indian Institute of Technology Ropar（印度理工学院Ropar分校）；Zhejiang University（浙江大学）；University of Oulu（奥卢大学）
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所）
Balancing methods for single-label data cannot be applied to multi-label problems as they would also resample the samples with high occurrences. We propose to reformulate this problem as an optimization problem in order to balance multi-label data. We apply this balancing algorithm to training datasets for detecting isolated facial movements, so-called Action Units. Several Action Units can describe combined emotions or physical states such as pain. As datasets in this area are limited and mostly imbalanced, we show how optimized balancing and then augmentation can improve Action Unit detectio…
