Human Reaction Intensity Estimation with Ensemble of Multi-task Networks

基于Transformer的视频中情绪反应强度估计与表情分类的多模态特征提取与融合

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

本文针对ABAW 2023的表情分类与情绪反应强度估计两项子挑战，提出利用多模型从Hume-Reaction数据集提取并融合音视频多模态特征的方法，在ERI任务上较基线提升84%。

Bootstrap Model Ensemble and Rank Loss for Engagement Intensity Regression

This paper presents our approach for the engagement intensity regression task of EmotiW 2019. The task is to predict the engagement intensity value of a student when he or she is watching an online MOOCs video in various conditions. Based on our winner solution last year, we mainly explore head features and body features with a bootstrap strategy and two novel loss functions in this paper. We maintain the framework of multi-instance learning with long short-term memory (LSTM) network, and make three contributions. First, besides of the gaze and head pose features, we explore facial landmark fe…

Emotional Reaction Intensity Estimation Based on Multimodal Data

This paper introduces our method for the Emotional Reaction Intensity (ERI) Estimation Challenge, in CVPR 2023: 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Based on the multimodal data provided by the originazers, we extract acoustic and visual features with different pretrained models. The multimodal features are mixed together by Transformer Encoders with cross-modal attention mechnism. In this paper, 1. better features are extracted with the SOTA pretrained models. 2. Compared with the baseline, we improve the Pearson's Correlations Coefficient a lot. 3. …

自然场景下情绪反应强度的计算机视觉估计

Computer Vision Estimation of Emotion Reaction Intensity in the Wild

本文提出在ABAW竞赛的ERI估计挑战中，利用视觉及多模态深度神经网络预测情绪反应强度，最佳模型在Hume-Reaction测试集上取得0.4080平均皮尔逊相关系数，为迈向非离散情绪的生产级模型奠定基础。

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges

The fifth Affective Behavior Analysis in-the-wild (ABAW) Competition is part of the respective ABAW Workshop which will be held in conjunction with IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2023. The 5th ABAW Competition is a continuation of the Competitions held at ECCV 2022, IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 and CVPR 2017 Conferences, and is dedicated at automatically analyzing affect. For this year's Competition, we feature two corpora: i) an extended version of the Aff-Wild2 database and ii) the Hume-Reaction dataset. The former database is an audiovisual one of…

整合整体与局部信息以估计情绪反应强度

Integrating Holistic and Local Information to Estimate Emotional Reaction Intensity

提出一种结合视频和音频的多模态架构，利用整体与局部特征及Transformer回归标记估计情绪反应强度，在ABAW5挑战中取得显著优于基线的性能。