再论视听学习中的预训练
Revisiting Pre-training in Audio-Visual Learning
摘要:
预训练技术在提高模型在各种任务上的性能方面取得了巨大的成功,但在某些单峰情况下,它的性能比从头开始训练更差。这启发我们思考:预训练的模型在更复杂的多模态场景中是否总是有效的,特别是对于诸如音频和视觉的异构模态?我们发现答案是否定的。具体来说,我们探索了预训练模型对两种视听学习场景的影响:跨模态初始化和多模态联合学习。采用跨模态初始化方法时,由于Batchnorm参数异常而导致的“死通道”现象阻碍了模型容量的利用。因此,我们提出了自适应批量范数重新初始化(ABRi),以更好地利用目标任务的预训练模型的能力。在多模态联合学习中,我们发现一个强的预训练的单模态编码器会对另一个模态的编码器带来负面影响。为了缓解这个问题,我们引入了一个两阶段的融合调整策略,更好地利用预训练的知识,同时使单峰编码器与自适应掩蔽方法合作。实验结果表明,该方法可以进一步挖掘预训练模型的潜力,提高视听学习的性能。
英文摘要:
Pre-training technique has gained tremendous success in enhancing model performance on various tasks, but found to perform worse than training from scratch in some uni-modal situations. This inspires us to think: are the pre-trained models always effective in the more complex multi-modal scenario, especially for the heterogeneous modalities such as audio and visual ones? We find that the answer is No. Specifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning. When cross-modal initialization is applied, the phenomena of "dead channel" caused by abnormal Batchnorm parameters hinders the utilization of model capacity. Thus, we propose Adaptive Batchnorm Re-initialization (ABRi) to better exploit the capacity of pre-trained models for target tasks. In multi-modal joint learning, we find a strong pre-trained uni-modal encoder would bring negative effects on the encoder of another modality. To alleviate such problem, we introduce a two-stage Fusion Tuning strategy, taking better advantage of the pre-trained knowledge while making the uni-modal encoders cooperate with an adaptive masking method. The experiment results show that our methods could further exploit pre-trained models' potential and boost performance in audio-visual learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。