arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.16918cs.LGcs.CV
收藏

m2mKD:面向模块化 Transformer 的模块到模块知识蒸馏

m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers

University of Macau(澳门大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Hong Kong(香港大学);Beijing University of Posts and Telecommunications(北京邮电大学);Peking University(北京大学);HKUST(香港科技大学)

Ka Man Lo, Yiming Liang, Wenyu Du, Yuantao Fan, Zili Wang, Wenhao Huang, Lei Ma, Jie Fu

AI总结:

本文提出模块到模块知识蒸馏(m2mKD),通过共享元模型连接单体教师模块与模块化学生模块,解决模块化 Transformer 训练优化难题,并在 NACs 和 V-MoE 上显著提升准确率与鲁棒性。

摘要:

模块化神经架构因其强大的泛化能力和对新领域的高效适应性而受到越来越多的关注。然而,由于内在稀疏连接带来的优化困难,训练这些模型面临挑战。利用知识蒸馏等技术从单体模型中获取知识,可以促进训练并实现多样化知识的整合。然而,传统的知识蒸馏方法并非为模块化模型量身定制,难以应对其独特的架构和庞大的参数规模。受这些挑战的启发,我们提出了模块到模块知识蒸馏(m2mKD),用于在模块之间传递知识。m2mKD 将预训练单体模型的教师模块和模块化模型的学生模块分别与一个共享元模型相结合,以促使学生模块模仿教师模块的行为。我们在两种模块化神经架构上评估了 m2mKD:神经注意力电路(NACs)和视觉混合专家(V-MoE)。将 m2mKD 应用于 NACs 在 Tiny-ImageNet 上的分布内(IID)准确率显著提升(最高达 5.6%),在 Tiny-ImageNet-R 上的分布外(OOD)鲁棒性也显著提升(最高达 4.2%)。此外,使用 m2mKD 训练的 V-MoE-Base 模型在 ImageNet-1k 上的准确率比端到端训练高出 3.5%。代码可在 https://github.com/kamanphoebe/m2mKD 获取。

英文摘要:

Modular neural architectures are gaining attention for their powerful generalization and efficient adaptation to new domains. However, training these models poses challenges due to optimization difficulties arising from intrinsic sparse connectivity. Leveraging knowledge from monolithic models through techniques like knowledge distillation can facilitate training and enable integration of diverse knowledge. Nevertheless, conventional knowledge distillation approaches are not tailored to modular models and struggle with unique architectures and enormous parameter counts. Motivated by these challenges, we propose module-to-module knowledge distillation (m2mKD) for transferring knowledge between modules. m2mKD combines teacher modules of a pretrained monolithic model and student modules of a modular model with a shared meta model respectively to encourage the student module to mimic the behaviour of the teacher module. We evaluate m2mKD on two modular neural architectures: Neural Attentive Circuits (NACs) and Vision Mixture-of-Experts (V-MoE). Applying m2mKD to NACs yields significant improvements in IID accuracy on Tiny-ImageNet (up to 5.6%) and OOD robustness on Tiny-ImageNet-R (up to 4.2%). Additionally, the V-MoE-Base model trained with m2mKD achieves 3.5% higher accuracy than end-to-end training on ImageNet-1k. Code is available at https://github.com/kamanphoebe/m2mKD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑