We tackle a challenging task: multi-view and multi-modal event detection that detects events in a wide-range real environment by utilizing data from distributed cameras and microphones and their weak labels. In this task, distributed sensors are utilized complementarily to capture events that are difficult to capture with a single sensor, such as a series of actions of people moving in an intricate room, or communication between people located far apart in a room. For sensors to cooperate effectively in such a situation, the system should be able to exchange information among sensors and combines information that is useful for identifying events in a complementary manner. For such a mechanism, we propose a Transformer-based multi-sensor fusion (MultiTrans) which combines multi-sensor data on the basis of the relationships between features of different viewpoints and modalities. In the experiments using a dataset newly collected for this task, our proposed method using MultiTrans improved the event detection performance and outperformed comparatives.
情感分析主要基于文本,近十年来发展迅速,引起了学术界和工业界的广泛关注。然而,现实世界中的信息通常来自多种形式,如音频和文本。因此,本文基于音频和文本,考虑多模态情感分析的任务,提出了一种新的融合策略,包括多特征融合和多模态融合,以提高音频-文本情感分析的准确性。我们称之为DFF-ATMF(Deep Feature Fusion - Audio and Text Modality Fusion)模型,它由两个并行的分支组成,即基于音频模态的分支和基于文本模态的分支。其核心机制是多特征向量和多通道注意的融合。在CMU-MOSI数据集和最近发布的CMU-MOSEI数据集上的实验,都是从YouTube上收集的情感分析,显示了我们的DFF-ATMF模型非常有竞争力的结果。此外,凭借注意力权重分布热图,我们还证明了使用DFF-ATMF学习的深度特征是互补的,并且是鲁棒的。令人惊讶的是,DFF-ATMF在IEMOCAP数据集上也取得了新的最先进的结果,表明所提出的融合策略对于多模态情感识别也具有良好的泛化能力。
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Bremen(不来梅大学);National University of Singapore(新加坡国立大学)
Multi-modal Automated Speech Scoring using Attention Fusion
Indraprastha Institute of Information Technology(英德拉普拉斯塔信息技术学院);Netaji Subhas Institute of Technology(内塔吉·苏巴斯理工学院);Teachers College, Columbia University(哥伦比亚大学教师学院);Second Language Testing Inc.(第二语言测试公司)
Sync-TVA: A Graph-Attention Framework for Multimodal Emotion Recognition with Cross-Modal Fusion
James Watt School of Engineering, University of Glasgow(格拉斯哥大学詹姆斯·瓦特工程学院);University of Bristol(布里斯托大学);School of Engineering Mathematics and Technology, University of Bristol(布里斯托大学工程数学与技术学院);School of Computing Science, University of Glasgow(格拉斯哥大学计算科学学院);Department of Civil, Environmental & Geomatic Engineering, University College London (UCL)(伦敦大学学院(UCL)土木、环境与测绘工程系)
Multimodal Audio-based Disease Prediction with Transformer-based Hierarchical Fusion Network
Purdue University(普渡大学);Beijing University of Chemical Technology(北京化工大学);New York University(纽约大学);University of Cincinnati(辛辛那提大学);Indiana University(印第安纳大学)