KDC-MAE：知识蒸馏对比掩码自动编码器
KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder
AI总结：
本文提出KDC-MAE，通过互补掩码策略和加权方式协同结合对比学习、知识蒸馏与掩码建模，在多个任务和多种模态上提升了自监督学习效果。
摘要：
在这项工作中，我们尝试扩展思路并展示自监督学习（SSL）学习范式的一种前进方向，通过结合对比学习、自蒸馏（知识蒸馏）和掩码数据建模这三种主要的SSL框架，来学习一种联合且协调的表示。所提出的SSL技术借助不同SSL学习目标的协作力量进行学习。因此，为了联合学习不同的SSL目标，我们提出了一种新的SSL架构KDC-MAE、一种用于学习模块对应关系的互补掩码策略，以及一种以协调方式将它们组合起来的加权方法。实验结果表明，对比掩码对应关系与知识蒸馏学习目标相结合，有助于在多个任务上对多种模态实现更好的学习。
英文摘要：
In this work, we attempted to extend the thought and showcase a way forward for the Self-supervised Learning (SSL) learning paradigm by combining contrastive learning, self-distillation (knowledge distillation) and masked data modelling, the three major SSL frameworks, to learn a joint and coordinated representation. The proposed technique of SSL learns by the collaborative power of different learning objectives of SSL. Hence to jointly learn the different SSL objectives we proposed a new SSL architecture KDC-MAE, a complementary masking strategy to learn the modular correspondence, and a weighted way to combine them coordinately. Experimental results conclude that the contrastive masking correspondence along with the KD learning objective has lent a hand to performing better learning for multiple modalities over multiple tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
