First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis
Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Computer Vision Center（计算机视觉中心）；Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition and Behaviour（唐德斯大脑、认知与行为研究所）；University of Barcelona（巴塞罗那大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；UPSud/INRIA Université Paris-Saclay（巴黎-萨克雷大学（巴黎第十一大学/法国国家信息与自动化研究所））；ChaLearn
Julio C. S. Jacques Junior, Yağmur Güçlütürk, Marc Pérez, Umut Güçlü, Carlos Andujar, Xavier Baró, Hugo Jair Escalante, Isabelle Guyon, Marcel A. J. van Gerven,… 展开作者
Julio C. S. Jacques Junior, Yağmur Güçlütürk, Marc Pérez, Umut Güçlü, Carlos Andujar, Xavier Baró, Hugo Jair Escalante, Isabelle Guyon, Marcel A. J. van Gerven, Rob van Lier, Sergio Escalera
备注：
Accepted on IEEE Transactions on Affective Computing (TAC)
Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use these information to infer apparent personality traits. Because of the overwhelming research interest in this topic, and of the potential impact that this sort of methods could have in society, we present in this paper an up-to-date review of existing vision-based approaches for apparent personality trait recognition. We describe seminal and cutting edge works on the subject, discussing and comparing their distinctive features and limitations. Future venues of research in the field are identified and discussed. Furthermore, aspects on the subjectivity in data labeling/evaluation, as well as current datasets and challenges organized to push the research on the field are reviewed.
Here, we develop an audiovisual deep residual network for multimodal apparent personality trait recognition. The network is trained end-to-end for predicting the Big Five personality traits of people from their videos. That is, the network does not require any feature engineering or visual analysis such as face detection, face landmark alignment or facial expression recognition. Recently, the network won the third place in the ChaLearn First Impressions Challenge with a test accuracy of 0.9109.
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
We propose a novel approach for First Impressions Recognition in terms of the Big Five personality-traits from short videos. The Big Five personality traits is a model to describe human personality using five broad categories: Extraversion, Agreeableness, Conscientiousness, Neuroticism and Openness. We train two bi-modal end-to-end deep neural network architectures using temporally ordered audio and novel stochastic visual features from few frames, without over-fitting. We empirically show that the trained models perform exceptionally well, even after training from a small sub-portions of inpu…
Shanghai Technical Institute of Electronics and Information（上海电子信息职业技术学院）
Human personality decides various aspects of their daily life and working behaviors. Since personality traits are relatively stable over time and unique for each subject, previous approaches frequently infer personality from a single frame or short-term behaviors. Moreover, most of them failed to specifically extract person-specific and unique cues for personality recognition. In this paper, we propose a novel video-based automatic personality traits recognition approach which consists of: (1) a \textbf{domain-specific facial behavior modelling} module that extracts personality-related multi-s…
Indian Institute of Technology Ropar（印度理工学院罗巴尔分校）；University of Canberra（堪培拉大学）
While personality traits have been extensively modeled as behavioral constructs, we model \textbf{\textit{job hirability}} as a \emph{personality construct}. On the {\emph{First Impressions Candidate Screening}} (FICS) dataset, we examine relationships among personality and hirability measures. Modeling hirability as a discrete/continuous variable with the \emph{big-five} personality traits as predictors, we utilize (a) apparent personality annotations, and (b) personality estimates obtained via audio, visual and textual cues for hirability prediction (HP). We also examine the efficacy of a tw…
Shanghai Technical Institute of Electronics and Information（上海电子信息职业技术学院）
Recent studies show that apparent personality traits can be reflected from human facial behavior dynamics. However, most existing methods can only encode single-scale short-term facial behaviors in the latent features for personality recognition. In this paper, we propose to recognize apparent personality recognition approach which first trains a person-specific network for each subject, modelling multi-scale long-term person-specific behavior evolution of the subject. Consequently, we hypothesize that the weights of the network contain the person-specific facial behavior-related cues of the s…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiamen University of Technology（厦门理工学院）
The human behavior of evaluating other individuals with respect to their personality traits and intelligence by evaluating their faces plays a crucial role in human relations. These trait judgments might influence important social outcomes in our lives such as elections and court sentences. Previous studies have reported that human can make valid inferences for at least four personality traits. In addition, some studies have demonstrated that facial trait evaluation can be learned using machine learning methods accurately. In this work, we experimentally explore whether self-reported personali…
University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；UPSud/INRIA, Université Paris-Saclay（巴黎萨克雷大学巴黎南大学/法国国家信息与自动化研究所）；ChaLearn（ChaLearn挑战学习组织）；Aix Marseille Univ（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）；Microsoft Research（微软研究院）；Boğaziçi University（博阿齐奇大学）；INAOE（墨西哥国家天体物理光学与电子研究所）；Namık Kemal University（纳默克·凯马尔大学）；Nagoya University（名古屋大学）；Radboud University（拉德堡德大学）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）
Explainability and interpretability are two critical aspects of decision support systems. Within computer vision, they are critical in certain tasks related to human behavior analysis such as in health care applications. Despite their importance, it is only recently that researchers are starting to explore these aspects. This paper provides an introduction to explainability and interpretability in the context of computer vision with an emphasis on looking at people tasks. Specifically, we review and study those mechanisms in the context of first impressions analysis. To the best of our knowled…
This paper presents a methodology to detect personality and basic emotion characteristics of crowds in video sequences. Firstly, individuals are detected and tracked, then groups are recognized and characterized. Such information is then mapped to OCEAN dimensions, used to find out personality and emotion in videos, based on OCC emotion models. Although it is a clear challenge to validate our results with real life experiments, we evaluate our method with the available literature information regarding OCEAN values of different Countries and also emergent Personal distance among people. Hence, …
Personality computing and affective computing, where the recognition of personality traits is essential, have gained increasing interest and attention in many research areas recently. We propose a novel approach to recognize the Big Five personality traits of people from videos. Personality and emotion affect the speaking style, facial expressions, body movements, and linguistic factors in social contexts, and they are affected by environmental elements. We develop a multimodal system to recognize apparent personality based on various modalities such as the face, environment, audio, and transc…
Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）
Personality perception is implicitly biased due to many subjective factors, such as cultural, social, contextual, gender and appearance. Approaches developed for automatic personality perception are not expected to predict the real personality of the target, but the personality external observers attributed to it. Hence, they have to deal with human bias, inherently transferred to the training data. However, bias analysis in personality computing is an almost unexplored area. In this work, we study different possible sources of bias affecting personality perception, including emotions from fac…
Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment
Hefei University of Technology（合肥工业大学）；Intelligent Interconnected Systems Laboratory of Anhui Province（安徽省智能互联系统实验室）；Jianghuai Advanced Technology Center（江淮前沿技术中心）；Anhui Provincial Industry Innovation Center of Humanoid Robots（安徽省人形机器人产业创新中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）
提出Traits Run Deeper框架，通过多模态基础表示、特质特异性非对称融合和分布校准回归模块，解决人格评估中模态偏好差异和标签偏差问题，在AVI Challenge 2026上MSE降低约25%。
Laboratoire d’imagerie, de vision et d’intelligence artificielle（成像、视觉与人工智能实验室）；École de technologie supérieure（高等工程技术学院）
Several methods have recently been proposed to analyze speech and automatically infer the personality of the speaker. These methods often rely on prosodic and other hand crafted speech processing features extracted with off-the-shelf toolboxes. To achieve high accuracy, numerous features are typically extracted using complex and highly parameterized algorithms. In this paper, a new method based on feature learning and spectrogram analysis is proposed to simplify the feature extraction process while maintaining a high level of accuracy. The proposed method learns a dictionary of discriminant fe…
Social media, as a major platform for communication and information exchange, is a rich repository of the opinions and sentiments of 2.3 billion users about a vast spectrum of topics. To sense the whys of certain social user's demands and cultural-driven interests, however, the knowledge embedded in the 1.8 billion pictures which are uploaded daily in public profiles has just started to be exploited since this process has been typically been text-based. Following this trend on visual-based social analysis, we present a novel methodology based on Deep Learning to build a combined image-and-text…
We present a generative adversarial learning approach to synthesize gaze behavior of a given personality. We train the model using an existing data set that comprises eye-tracking data and personality traits of 42 participants performing an everyday task. Given the values of Big-Five personality traits (openness, conscientiousness, extroversion, agreeableness, and neuroticism), our model generates time series data consisting of gaze target, blinking times, and pupil dimensions. We use the generated data to synthesize the gaze motion of virtual agents on a game engine.
VinAI Research（VinAI研究院）；University of Information Technology, VNU-HCM（越南国家大学胡志明市分校信息科技大学）；Stony Brook University（石溪大学）；University of Oregon（俄勒冈大学）
Personality image captioning (PIC) aims to describe an image with a natural language caption given a personality trait. In this work, we introduce a novel formulation for PIC based on a communication game between a speaker and a listener. The speaker attempts to generate natural language captions while the listener encourages the generated captions to contain discriminative information about the input images and personality traits. In this way, we expect that the generated captions can be improved to naturally represent the images and express the traits. In addition, we propose to adapt the la…
California Institute of Technology（加州理工学院）；University of Trento（特伦托大学）；CIMEC - University of Trento（心智/脑科学中心 - 特伦托大学）；Stanford University（斯坦福大学）；University of Cambridge（剑桥大学）；University of Verona（维罗纳大学）；FBK（布鲁诺·凯斯勒基金会）
People spend considerable effort managing the impressions they give others. Social psychologists have shown that people manage these impressions differently depending upon their personality. Facebook and other social media provide a new forum for this fundamental process; hence, understanding people's behaviour on social media could provide interesting insights on their personality. In this paper we investigate automatic personality recognition from Facebook profile pictures. We analyze the effectiveness of four families of visual features and we discuss some human interpretable patterns that …
