首页/论文/2012.00063

arXiv 2012.000632020-12eess.AScs.SDeess.IV

Detecting expressions with multimodal transformers

Amazon（亚马逊）

Srinivas Parthasarathy, Shiva Sundaram

备注：: IEEE Spoken Language Technology Workshop 2021

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Developing machine learning algorithms to understand person-to-person engagement can result in natural user experiences for communal devices such as Amazon Alexa. Among other cues such as voice activity and gaze, a person's audio-visual expression that includes tone of the voice and facial expression serves as an implicit signal of engagement between parties in a dialog. This study investigates deep-learning algorithms for audio-visual detection of user's expression. We first implement an audio-visual baseline model with recurrent layers that shows competitive results compared to current state of the art. Next, we propose the transformer architecture with encoder layers that better integrate audio-visual features for expressions tracking. Performance on the Aff-Wild2 database shows that the proposed methods perform better than baseline architecture with recurrent layers with absolute gains approximately 2% for arousal and valence descriptors. Further, multimodal architectures show significant improvements over models trained on single modalities with gains of up to 3.6%. Ablation studies show the significance of the visual modality for the expression detection on the Aff-Wild2 database.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

多模态Transformer蒸馏用于音视频同步

Multimodal Transformer Distillation for Audio-Visual Synchronization

National Taiwan University（台湾大学）

本文提出MTDVocaLiST模型，通过多模态Transformer蒸馏损失和不确定性加权，在显著缩小模型规模（减少83.52%）的同时，性能超越同规模SOTA模型，实现高效音视频同步。

2024-03-18cs.CVcs.IRcs.SD

Key-Sparse Transformer for Multimodal Speech Emotion Recognition

South China University of Technology（华南理工大学）；UBTECH Robotics Corp（优必选机器人公司）

Speech emotion recognition is a challenging research topic that plays a critical role in human-computer interaction. Multimodal inputs further improve the performance as more emotional information is used. However, existing studies learn all the information in the sample while only a small portion of it is about emotion. The redundant information will become noises and limit the system performance. In this paper, a key-sparse Transformer is proposed for efficient emotion recognition by focusing more on emotion related information. The proposed method is evaluated on the IEMOCAP and LSSED. Expe…

2023-02-27cs.SDeess.AS

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned…

2021-12-03cs.CLcs.CVcs.LG

面向对话情绪识别的融合混合对比学习的多模态提示Transformer

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

Chongqing University of Technology（重庆理工大学）

针对多模态对话情绪识别中跨模态融合噪声和少样本标签预测难题，提出融合混合对比学习的多模态提示Transformer，通过提示增强与分层融合及混合对比优化，在两个基准数据集上取得SOTA表现。

2023-10-04cs.CLcs.SDeess.AS

用于音视频分类的高效选择性音频掩码多模态瓶颈Transformer

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

本文提出了一种名为AVT的音视频识别方法，通过瓶颈Transformer降低跨模态融合复杂度，并结合自监督目标与掩码音频片段损失提升学习效率，在多个数据集上显著超越现有方法。

2024-01-08cs.AIcs.CVcs.LG

使用多模态双注意力 Transformer 的跨语言语音情感识别

Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers

Qatar University（卡塔尔大学）

针对跨语言语音情感识别性能受限的问题，提出多模态双注意力 Transformer（MDAT），融合图注意力、协同注意力与 transformer 编码，在少量目标语言数据下提升分类效果。

2023-07-14cs.CLcs.SDeess.AS

更多相关论文收起更多论文

Multiresolution and Multimodal Speech Recognition with Transformers

Amazon Lab126（亚马逊Lab126）

This paper presents an audio visual automatic speech recognition (AV-ASR) system using a Transformer-based architecture. We particularly focus on the scene context provided by the visual information, to ground the ASR. We extract representations for audio features in the encoder layers of the transformer and fuse video features using an additional crossmodal multihead attention layer. Additionally, we incorporate a multitask training criterion for multiresolution ASR, where we train the model to generate both character and subword level transcriptions. Experimental results on the How2 dataset,…

2020-04-29cs.CVcs.LGcs.SD

MAQA: A Multimodal QA Benchmark for Negation

Google Research（谷歌研究院）；Seoul National University（首尔国立大学）

Multimodal learning can benefit from the representation power of pretrained Large Language Models (LLMs). However, state-of-the-art transformer based LLMs often ignore negations in natural language and there is no existing benchmark to quantitatively evaluate whether multimodal transformers inherit this weakness. In this study, we present a new multimodal question answering (QA) benchmark adapted from labeled music videos in AudioSet (Gemmeke et al., 2017) with the goal of systematically evaluating if multimodal transformers can perform complex reasoning to recognize new concepts as negation o…

2023-01-09cs.AIcs.CLcs.LG

用于音视频分类的高效多尺度多模态瓶颈Transformer

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

本文提出多尺度多模态Transformer（MMT），结合多尺度音频与视频Transformer及对比损失实现高效音视频分类，在多个基准上显著提升准确率并降低计算开销。

2024-01-08cs.AIcs.CVcs.LG

TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation

Didi Chuxing（滴滴出行）

Multimodalities provide promising performance than unimodality in most tasks. However, learning the semantic of the representations from multimodalities efficiently is extremely challenging. To tackle this, we propose the Transformer based Cross-modal Translator (TCT) to learn unimodal sequence representations by translating from other related multimodal sequences on a supervised learning method. Combined TCT with Multimodal Transformer Network (MTN), we evaluate MTN-TCT on the video-grounded dialogue which uses multimodality. The proposed method reports new state-of-the-art performance on vid…

2019-10-23cs.CLcs.CVcs.LG

MDD-Net：通过Mutual Transformer进行多模态抑郁症检测

MDD-Net: Multimodal Depression Detection through Mutual Transformer

Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo（模式分析与机器智能中心，电气与计算机工程系，滑铁卢大学）；School of Engineering and Computing, Department of Computer Science and Engineering, American University of Ras Al Khaimah（工程与计算学院，计算机科学与工程系，阿联酋拉线哈姆市美国大学）

本研究提出MDD-Net，利用mutual transformer融合声学和视觉特征进行抑郁症检测，在D-Vlog数据集上F1-Score较现有最优方法提升最高达17.37%。

2025-08-11cs.CVcs.LGcs.MM