arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.04385cs.LG
收藏

多模态学习中的鲁棒性研究

On Robustness in Multimodal Learning

Apple(苹果公司);Apple ML Research(苹果机器学习研究部门)

Brandon McKinzie, Joseph Cheng, Vaishaal Shankar, Yinfei Yang, Jonathon Shlens, Alexander Toshev

AI总结:

本文提出多模态鲁棒性框架,系统分析常见多模态表示学习方法在模态分布变化下的不足,并通过两种干预技术在三个数据集上实现1.5至4倍鲁棒性提升。

摘要:

多模态学习被定义为对视频、音频和文本等多种异构输入模态进行学习。在这项工作中,我们关注的是理解模型在训练和部署之间模态类型不同的情况下如何表现,这种情况在多模态学习应用于硬件平台的许多场景中自然出现。我们提出了一个多模态鲁棒性框架,以对常见的多模态表示学习方法进行系统性分析。此外,我们识别了这些方法在鲁棒性方面的不足,并提出了两种干预技术,在 AudioSet、Kinetics-400 和 ImageNet-Captions 三个数据集上带来了 1.5 倍至 4 倍的鲁棒性提升。最后,我们证明这些干预措施能够更好地利用额外模态(如果存在),在 AudioSet 20K 上取得了 44.2 mAP 的竞争性结果。

英文摘要:

Multimodal learning is defined as learning over multiple heterogeneous input modalities such as video, audio, and text. In this work, we are concerned with understanding how models behave as the type of modalities differ between training and deployment, a situation that naturally arises in many applications of multimodal learning to hardware platforms. We present a multimodal robustness framework to provide a systematic analysis of common multimodal representation learning methods. Further, we identify robustness short-comings of these approaches and propose two intervention techniques leading to $1.5\times$-$4\times$ robustness improvements on three datasets, AudioSet, Kinetics-400 and ImageNet-Captions. Finally, we demonstrate that these interventions better utilize additional modalities, if present, to achieve competitive results of $44.2$ mAP on AudioSet 20K.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑