Multimodal learning is defined as learning over multiple heterogeneous input modalities such as video, audio, and text. In this work, we are concerned with understanding how models behave as the type of modalities differ between training and deployment, a situation that naturally arises in many applications of multimodal learning to hardware platforms. We present a multimodal robustness framework to provide a systematic analysis of common multimodal representation learning methods. Further, we identify robustness short-comings of these approaches and propose two intervention techniques leading to $1.5\times$-$4\times$ robustness improvements on three datasets, AudioSet, Kinetics-400 and ImageNet-Captions. Finally, we demonstrate that these interventions better utilize additional modalities, if present, to achieve competitive results of $44.2$ mAP on AudioSet 20K.
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Columbia University(哥伦比亚大学);Google(谷歌);Cornell University(康奈尔大学)
我们提出了一个框架,学习多模态表示从未标记的数据使用卷积自由Transformer架构。具体来说,我们的视频-音频-文本Transformer(VATT)将原始信号作为输入,并提取多模态表示,这些表示足够丰富,可以使各种下游任务受益。我们使用多模态对比损失从头开始训练VATT端到端,并通过视频动作识别、音频事件分类、图像分类和文本到视频检索等下游任务来评估其性能。此外,我们研究了一个模态不可知的,单骨干Transformer共享三个模态之间的权重。我们证明了无卷积VATT在下游任务中的性能优于最先进的基于ConvNet的架构。特别是,VATT的视觉Transformer在Kinetics-400上达到了82.1%的顶级准确率,在Kinetics-600上达到了83.6%,在Kinetics-700上达到了72.7%,在Moments in Time上达到了41.1%,这些都是新的记录,同时避免了监督式的预训练。转移到图像分类导致ImageNet上的top-1准确率为78.7%,而从头开始训练相同的Transformer的准确率为64.7%,这表明尽管视频和图像之间存在领域差距,但我们的模型具有普遍性。VATT的音频Transformer还在基于波形的音频事件识别方面创造了新纪录,在AudioSet上实现了39.4%的mAP,而无需任何监督的预训练。VATT的源代码是公开的。
Look, Listen and Learn - A Multimodal LSTM for Speaker Identification
SenseTime Group Limited(商汤集团有限公司);The University of Hong Kong(香港大学)
说话人辨识是指定位与视频中正在进行的语音具有相同身份的人的面部的任务。这项任务不仅需要对视觉和听觉信号进行集体感知,而且处理严重质量下降和无约束内容变化的鲁棒性也是必不可少的。在本文中,我们描述了一种新的多模态长短期记忆(LSTM)架构,它从每个序列输入的开始就无缝地统一了视觉和听觉模态。其关键思想是扩展传统的LSTM,不仅在时间步上共享权重,而且在模态上共享权重。我们表明,在面部和语音的时间依赖性建模可以显着提高内容质量下降和变化的鲁棒性。我们还发现,我们的多模态LSTM对干扰物(即非说话身份)具有鲁棒性。我们将我们的多模态LSTM应用于The Big Bang Theory数据集,并表明我们的系统在说话人辨识方面优于最先进的系统,具有更低的误报率和更高的识别准确率。
RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants
Singapore University of Technology and Design(新加坡科技设计大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Nanyang Technological University(南洋理工大学);Chongqing University(重庆大学)
Buffer replay enhances the robustness of multimodal learning under missing-modality
College of Computer Science and Electronic Engineering Hunan University(湖南大学计算机科学与电子工程学院);School of Computer Science Central South University(中南大学计算机学院)
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
Institute of Automation, Chinese Academy of Sciences (CAS)(中国科学院自动化研究所);Renmin University of China(中国人民大学);Shanghai Jiao Tong University(上海交通大学);Inner Mongolia University(内蒙古大学);National University of Defense Technology(国防科技大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Imperial College London(帝国理工学院);Tsinghua University(清华大学)
Deep Multimodal Learning with Missing Modality: A Survey
The Australian National University(澳大利亚国立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Adelaide University(阿德莱德大学);The University of Surrey(萨里大学)