arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06244cs.CVcs.MM
收藏

量化并增强多模态鲁棒性：基于模态偏好

Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）

Zequn Yang, Yake Wei, Ce Liang, Di Hu

备注：
Accepted to ICLR 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过理论分析发现模态偏好影响多模态鲁棒性，提出可认证鲁棒多模态训练（CRMT）方法，通过调节基本组件显著提升鲁棒性，且易于扩展。

摘要：

多模态模型在有效整合来自不同来源的信息方面展现出良好的能力，然而，它们被发现容易受到普遍存在的扰动的影响，例如单模态攻击和缺失条件。为了应对这些扰动，期望获得鲁棒的多模态表示，这些表示应远离判别性的多模态决策边界。在本文中，不同于传统的实证研究，我们关注一个常用的联合多模态框架，并从理论上发现，更大的单模态表示间隔和更可靠的模态集成是实现更高鲁棒性的基本组成部分。这一发现可以进一步解释多模态鲁棒性的局限性以及多模态模型常常容易受到针对特定模态攻击的现象。此外，我们的分析揭示了模型对模态具有不同偏好的普遍问题如何通过影响这些基本组成部分来限制多模态鲁棒性，并可能导致针对特定模态的攻击非常有效。受我们理论发现的启发，我们引入了一种称为可认证鲁棒多模态训练（CRMT）的训练程序，它可以减轻模态偏好的影响，并明确调节基本组成部分，以可认证的方式显著提高鲁棒性。与现有方法相比，我们的方法在性能和鲁棒性方面展示了显著的改进。此外，我们的训练程序可以轻松扩展以增强其他鲁棒训练策略，突显了其可信性和灵活性。

英文摘要：

Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑