面向数字病理学的多教师知识蒸馏扩展
Scaling Multi-Teacher Distillation for Digital Pathology
AI总结：
本文针对多教师知识蒸馏扩展问题，提出含可学习教师令牌等的改进方法，训练出效率提升10倍的OMNI模型，在39项数字病理学任务上性能优于或匹配教师模型。
摘要：
多教师知识蒸馏（Multi-teacher distillation）可将多个“教师”网络的知识迁移至单个“学生”网络，该方法在数字病理学等领域颇具应用前景，近期该领域已提出诸多强大的基础模型。然而，本文研究表明，标准多教师知识蒸馏方法在用于训练学生编码器的教师数量增加时表现不佳。我们证实，学习教师特定表示是多教师知识蒸馏（MTD）实现扩展的关键。重要的是，我们提出了不同的设计选择，包括可学习教师令牌（learnable teacher tokens）、定制化注意力机制、额外的混合专家（mixture-of-experts）层以及对比损失，以更好地学习此类教师特定表示。我们的方法在教师数量增加时扩展性更优，可训练出紧凑的学生编码器，其效率比更大的教师基础模型高10倍，同时性能与教师模型相当，甚至在大量图块级（tile-level）和切片级（slide-level）任务上表现更优。我们开展了全面的实证验证，在涵盖图块级和切片级的39项任务上评估了10余个基础模型。最终，我们发布了新的强大高效基础模型集合OMNI，该模型由10个最先进基础模型的知识训练而成。我们最后对学习到的教师特定表示进行了分析，强调了它们的互补性，并解释了为何这些表示可在下游任务阶段轻松聚合。
英文摘要：
Multi-teacher distillation allows transferring knowledge from multiple "teacher" networks to a single "student" network. This method is promising in fields such as digital pathology, where many powerful foundation models were proposed recently. However, as we show in this paper, the standard multi-teacher distillation approach reacts poorly to an increase in the number of teachers used to train a student encoder. We demonstrate that learning teacher-specific representations is a key to scaling in MTD. Importantly, different design choices, such as learnable teacher tokens, a tailored attention scheme, additional mixture-of-experts layers and a contrastive loss, are proposed to better learn such teacher-specific representations. We show that our method better scales with respect to the number of considered teachers, allowing us to train compact student encoders, up to 10x more efficient than larger teacher foundation models, while matching their performance and even outperforming them on a large set of tile-level and slide-level tasks. We conduct a thorough empirical validation, evaluating more than 10 foundation models on 39 tasks spanning tile and slide levels. As a result, we release a new collection of strong and efficient foundation models, named OMNI, trained from the knowledge of 10 state-of-the-art foundation models. We finally conduct an analysis of the learned teacher-specific representations, highlighting their complementarity and explaining why they can be easily aggregated at downstream time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
