arXiv 2405.093212024-05-15cs.CVcs.AIcs.LGcs.MM

ReconBoost：Boosting 可实现模态协调

ReconBoost: Boosting Can Achieve Modality Reconcilement

Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences（中国科学院大数据挖掘与知识管理重点实验室）

Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, Qingming Huang

备注：: This paper has been accepted by ICML2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态学习中模态竞争问题，提出基于交替学习的 ReconBoost 方法，结合类梯度提升机制、记忆巩固与全局校正方案，在六个基准上验证了有效性。

摘要：

本文探索了一种新颖的多模态交替学习范式，旨在在单模态特征的利用与跨模态交互的探索之间实现协调。研究动机源于当前多模态学习范式倾向于同时探索多模态特征这一事实，由此产生的梯度会阻碍弱模态中特征的进一步挖掘，引发模态竞争——即主导模态会压制整个学习过程。为解决这一问题，本文研究模态交替学习范式以实现模态协调。具体而言，我们提出了名为 ReconBoost 的新方法，每次仅更新一个固定模态。该方法的学习目标会通过协调正则项动态调整，以避免与历史模型产生竞争。通过选用基于 KL 散度的协调机制，我们证明所提方法与 Friedman 的 Gradient-Boosting（GB，梯度提升）算法具有相似性：更新后的学习器能够修正其他学习器产生的误差，助力提升整体性能。与经典 GB 算法的主要区别在于，我们仅为每个模态保留最新模型，避免因集成强学习器而导致过拟合。此外，我们还提出了记忆巩固方案和全局校正方案，以进一步提升该策略的有效性。在六个多模态基准数据集上的实验验证了该方法的有效性。我们已将代码发布于 https://github.com/huacong/ReconBoost。

英文摘要：

This paper explores a novel multi-modal alternating learning paradigm pursuing a reconciliation between the exploitation of uni-modal features and the exploration of cross-modal interactions. This is motivated by the fact that current paradigms of multi-modal learning tend to explore multi-modal features simultaneously. The resulting gradient prohibits further exploitation of the features in the weak modality, leading to modality competition, where the dominant modality overpowers the learning process. To address this issue, we study the modality-alternating learning paradigm to achieve reconcilement. Specifically, we propose a new method called ReconBoost to update a fixed modality each time. Herein, the learning objective is dynamically adjusted with a reconcilement regularization against competition with the historical models. By choosing a KL-based reconcilement, we show that the proposed method resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated learner can correct errors made by others and help enhance the overall performance. The major difference with the classic GB is that we only preserve the newest model for each modality to avoid overfitting caused by ensembling strong learners. Furthermore, we propose a memory consolidation scheme and a global rectification scheme to make this strategy more effective. Experiments over six multi-modal benchmarks speak to the efficacy of the method. We release the code at https://github.com/huacong/ReconBoost.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。