UNIC：通过多教师蒸馏实现通用分类模型
UNIC: Universal Classification Models via Multi-teacher Distillation
AI总结：
本文提出UNIC，通过分析并改进多教师蒸馏，利用可扩展投影器阶梯和teacher dropping训练单一通用分类编码器，使其在容量不变的情况下保持或超越各任务最佳教师。
摘要：
预训练模型已成为一种常用资源，并在广泛任务上提供强劲结果。在本工作中，我们聚焦分类，并寻求学习一个独特的编码器，使其能够吸收若干互补预训练模型的知识。我们的目标是在各种分类任务上实现更强的泛化。我们提出通过多教师蒸馏来学习这样的编码器。我们首先彻底分析由多个具有互补优势的强教师驱动时的标准蒸馏。在该分析的指导下，我们逐步提出对基础蒸馏设置的改进。其中，我们用一个可扩展投影器阶梯丰富编码器架构，从而在蒸馏过程中增强中间特征的影响；同时引入 teacher dropping（教师丢弃），这是一种能更好平衡各教师影响的正则化机制。我们最终的蒸馏策略产生的学生模型与任一教师具有相同容量，同时在每项任务上保持或提升最佳教师的性能。项目页面与代码：https://europe.naverlabs.com/unic
英文摘要：
Pretrained models have become a commodity and offer strong results on a broad range of tasks. In this work, we focus on classification and seek to learn a unique encoder able to take from several complementary pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers' influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task. Project page and code: https://europe.naverlabs.com/unic
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
