面向数字病理学的稳健高效模型的基础模型蒸馏
Distilling foundation models for robust and efficient models in digital pathology
AI总结：
本文通过将大型基础模型蒸馏为小型模型H0-mini，大幅减少参数并降低推理成本，在多个基准上取得优异排名，且对染色和扫描变化具有高稳健性，为数字病理学提供轻量高效方案。
摘要：
近年来，用于数字病理学的基础模型（FM）的出现严重依赖于扩大预训练数据集和模型规模，从而产生了庞大而强大的模型。虽然这提高了各种下游任务的性能，但也带来了计算成本和推理时间的增加。在这项工作中，我们探索将大型基础模型蒸馏为较小的模型，将参数数量减少几个数量级。利用蒸馏技术，我们的蒸馏模型H0-mini在显著降低推理成本的同时，实现了与大型基础模型几乎相当的性能。它在多个公共基准上进行了评估，在HEST基准上排名第3，在EVA基准上排名第5。此外，在PLISM数据集上进行的稳健性分析表明，我们的蒸馏模型对染色和扫描条件的变化具有出色的稳健性，显著优于其他最先进的模型。这为设计轻量级且稳健的数字病理学模型开辟了新的前景，同时不损害性能。
英文摘要：
In recent years, the advent of foundation models (FM) for digital pathology has relied heavily on scaling the pre-training datasets and the model size, yielding large and powerful models. While it resulted in improving the performance on diverse downstream tasks, it also introduced increased computational cost and inference time. In this work, we explore the distillation of a large foundation model into a smaller one, reducing the number of parameters by several orders of magnitude. Leveraging distillation techniques, our distilled model, H0-mini, achieves nearly comparable performance to large FMs at a significantly reduced inference cost. It is evaluated on several public benchmarks, achieving 3rd place on the HEST benchmark and 5th place on the EVA benchmark. Additionally, a robustness analysis conducted on the PLISM dataset demonstrates that our distilled model reaches excellent robustness to variations in staining and scanning conditions, significantly outperforming other state-of-the art models. This opens new perspectives to design lightweight and robust models for digital pathology, without compromising on performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
