Personality computing and affective computing, where the recognition of personality traits is essential, have gained increasing interest and attention in many research areas recently. We propose a novel approach to recognize the Big Five personality traits of people from videos. Personality and emotion affect the speaking style, facial expressions, body movements, and linguistic factors in social contexts, and they are affected by environmental elements. We develop a multimodal system to recognize apparent personality based on various modalities such as the face, environment, audio, and transcription features. We use modality-specific neural networks that learn to recognize the traits independently and we obtain a final prediction of apparent personality with a feature-level fusion of these networks. We employ pre-trained deep convolutional neural networks such as ResNet and VGGish networks to extract high-level features and Long Short-Term Memory networks to integrate temporal information. We train the large model consisting of modality-specific subnetworks using a two-stage training process. We first train the subnetworks separately and then fine-tune the overall model using these trained networks. We evaluate the proposed method using ChaLearn First Impressions V2 challenge dataset. Our approach obtains the best overall "mean accuracy" score, averaged over five personality traits, compared to the state-of-the-art.
Here, we develop an audiovisual deep residual network for multimodal apparent personality trait recognition. The network is trained end-to-end for predicting the Big Five personality traits of people from their videos. That is, the network does not require any feature engineering or visual analysis such as face detection, face landmark alignment or facial expression recognition. Recently, the network won the third place in the ChaLearn First Impressions Challenge with a test accuracy of 0.9109.
Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）
Personality perception is implicitly biased due to many subjective factors, such as cultural, social, contextual, gender and appearance. Approaches developed for automatic personality perception are not expected to predict the real personality of the target, but the personality external observers attributed to it. Hence, they have to deal with human bias, inherently transferred to the training data. However, bias analysis in personality computing is an almost unexplored area. In this work, we study different possible sources of bias affecting personality perception, including emotions from fac…
Shanghai Technical Institute of Electronics and Information（上海电子信息职业技术学院）
Recent studies show that apparent personality traits can be reflected from human facial behavior dynamics. However, most existing methods can only encode single-scale short-term facial behaviors in the latent features for personality recognition. In this paper, we propose to recognize apparent personality recognition approach which first trains a person-specific network for each subject, modelling multi-scale long-term person-specific behavior evolution of the subject. Consequently, we hypothesize that the weights of the network contain the person-specific facial behavior-related cues of the s…
Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；ETH Zurich（苏黎世联邦理工学院）；Max Planck ETH Center for Learning Systems（马克斯·普朗克-苏黎世联邦理工学院学习系统中心）；National University of Singapore (NUS)（新加坡国立大学）；Advanced Digital Sciences Center (ADSC), University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校高级数字科学中心）
Apparent personality and emotion analysis are both central to affective computing. Existing works solve them individually. In this paper we investigate if such high-level affect traits and their relationship can be jointly learned from face images in the wild. To this end, we introduce PersEmoN, an end-to-end trainable and deep Siamese-like network. It consists of two convolutional network branches, one for emotion and the other for apparent personality. Both networks share their bottom feature extraction module and are optimized within a multi-task learning framework. Emotion and personality …
Shanghai Technical Institute of Electronics and Information（上海电子信息职业技术学院）
Human personality decides various aspects of their daily life and working behaviors. Since personality traits are relatively stable over time and unique for each subject, previous approaches frequently infer personality from a single frame or short-term behaviors. Moreover, most of them failed to specifically extract person-specific and unique cues for personality recognition. In this paper, we propose a novel video-based automatic personality traits recognition approach which consists of: (1) a \textbf{domain-specific facial behavior modelling} module that extracts personality-related multi-s…
Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Computer Vision Center（计算机视觉中心）；Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition and Behaviour（唐德斯大脑、认知与行为研究所）；University of Barcelona（巴塞罗那大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；UPSud/INRIA Université Paris-Saclay（巴黎-萨克雷大学（巴黎第十一大学/法国国家信息与自动化研究所））；ChaLearn
Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use the…
Knowing You at First Glance: Inferring Apparent Personality from Faces
Institute of Automation, CAS（中国科学院自动化研究所）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）；School of Cyber Security, UCAS（中国科学院大学网络空间安全学院）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS（中国科学院香港创新研究院人工智能与机器人中心）；School of Computer Science and Engineering, the Faculty of Innovation Engineering, M.U.S.T（澳门科技大学创新工程学院计算机科学与工程系）
MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics
Beijing Institute of Technology（北京理工大学）；Xi’an Jiaotong Liverpool University（西安交通大学利物浦大学）；Institute of Automation, Chinese Academy of Sciences (CAS)（中国科学院自动化研究所）；Anhui University（安徽大学）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心，清华大学）；Department of Automation, Tsinghua University（清华大学自动化系）；ShanghaiTech University（上海科技大学）；University of Chinese Academy of Sciences（中国科学院大学）；Tianjin Normal University（天津师范大学）；Institute of Automation, CAS（中国科学院自动化研究所）；Institute of Psychology, CAS（中国科学院心理研究所）
Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset
Shanghai Center for Brain Science and Brain-Inspired Technology（上海脑科学与类脑研究中心）；NYU-ECNU Institute of Brain and Cognitive Science, New York University Shanghai（上海纽约大学纽约大学-华东师范大学脑与认知科学研究院）
Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment
Hefei University of Technology（合肥工业大学）；Intelligent Interconnected Systems Laboratory of Anhui Province（安徽省智能互联系统实验室）；Jianghuai Advanced Technology Center（江淮前沿技术中心）；Anhui Provincial Industry Innovation Center of Humanoid Robots（安徽省人形机器人产业创新中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）
提出Traits Run Deeper框架，通过多模态基础表示、特质特异性非对称融合和分布校准回归模块，解决人格评估中模态偏好差异和标签偏差问题，在AVI Challenge 2026上MSE降低约25%。
University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；UPSud/INRIA, Université Paris-Saclay（巴黎萨克雷大学巴黎南大学/法国国家信息与自动化研究所）；ChaLearn（ChaLearn挑战学习组织）；Aix Marseille Univ（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）；Microsoft Research（微软研究院）；Boğaziçi University（博阿齐奇大学）；INAOE（墨西哥国家天体物理光学与电子研究所）；Namık Kemal University（纳默克·凯马尔大学）；Nagoya University（名古屋大学）；Radboud University（拉德堡德大学）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）
Explainability and interpretability are two critical aspects of decision support systems. Within computer vision, they are critical in certain tasks related to human behavior analysis such as in health care applications. Despite their importance, it is only recently that researchers are starting to explore these aspects. This paper provides an introduction to explainability and interpretability in the context of computer vision with an emphasis on looking at people tasks. Specifically, we review and study those mechanisms in the context of first impressions analysis. To the best of our knowled…
INRIA（法国国家信息与自动化研究所）；Indian Institute of Information Technology, Allahabad（印度阿拉哈巴德信息技术学院）；Birla Institute of Technology and Science, Pilani（印度皮拉尼贝拉技术科学学院）
Personality computing and affective computing have gained recent interest in many research areas. The datasets for the task generally have multiple modalities like video, audio, language and bio-signals. In this paper, we propose a flexible model for the task which exploits all available data. The task involves complex relations and to avoid using a large model for video processing specifically, we propose the use of behaviour encoding which boosts performance with minimal change to the model. Cross-attention using transformers has become popular in recent times and is utilised for fusion of d…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
We propose a novel approach for First Impressions Recognition in terms of the Big Five personality-traits from short videos. The Big Five personality traits is a model to describe human personality using five broad categories: Extraversion, Agreeableness, Conscientiousness, Neuroticism and Openness. We train two bi-modal end-to-end deep neural network architectures using temporally ordered audio and novel stochastic visual features from few frames, without over-fitting. We empirically show that the trained models perform exceptionally well, even after training from a small sub-portions of inpu…
Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability
Technology Application and Human Resource Development, National Taiwan Normal University（台湾国立台中教育大学技术应用与人力资源发展系）；Computer Science and Information Engineering, National Central University（台湾国立中央大学计算机科学与资讯工程系）；Institute of Photonic System, National Yang Ming Chiao Tung University（台湾阳明交通大学光电系统研究所）
Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice
Numediart Institute, ISIA Lab, University of Mons（Numediart研究院、ISIA实验室、蒙斯大学）；Institute for Creative Technologies, University of Southern California（创意技术研究所、南加州大学）
Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）；University of Canberra（堪培拉大学）；University of Glasgow（格拉斯哥大学）
We explore the efficacy of multimodal behavioral cues for explainable prediction of personality and interview-specific traits. We utilize elementary head-motion units named kinemes, atomic facial movements termed action units and speech features to estimate these human-centered traits. Empirical results confirm that kinemes and action units enable discovery of multiple trait-specific behaviors while also enabling explainability in support of the predictions. For fusing cues, we explore decision and feature-level fusion, and an additive attention-based fusion strategy which quantifies the relat…
Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to co…
