arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.12913cs.LGcs.CVstat.ML
收藏

torchdistill:一个模块化的、解析驱动的知识蒸馏框架

torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

University of California, Irvine(加州大学尔湾分校)

Yoshitomo Matsubara

摘要:

虽然知识蒸馏(转移)一直吸引着研究界的关注,但该领域的最新发展提高了对可重复研究和高度通用框架的需求,以降低这种高质量,可重复的深度学习研究的障碍。一些研究人员自愿发表了他们在知识蒸馏研究中使用的框架,以帮助其他感兴趣的研究人员复制他们的原始工作。然而,这样的框架通常既不能很好地推广,也不能很好地维护,因此研究人员仍然需要编写大量的代码来重构/构建框架,以引入新的方法,模型,数据集和设计实验。在本文中,我们提出了我们开发的开源框架,建立在PyTorch和专用于知识蒸馏研究。该框架旨在使用户能够通过声明式PyYAML配置文件设计实验,并帮助研究人员完成最近提出的ML代码完整性检查表。使用开发的框架,我们展示了其各种有效的培训策略,并实现了各种知识蒸馏方法。我们还在ICLR、NeurIPS、CVPR和ECCV等主要机器学习会议上展示了他们在ImageNet和COCO数据集上的一些原始实验结果,包括最新的最先进的方法。所有源代码、配置、日志文件和训练模型权重都可以在https://github.com/yoshitomo-matsubara/torchdistill上公开获得。

英文摘要:

While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design experiments by declarative PyYAML configuration files, and helps researchers complete the recently proposed ML Code Completeness Checklist. Using the developed framework, we demonstrate its various efficient training strategies, and implement a variety of knowledge distillation methods. We also reproduce some of their original experimental results on the ImageNet and COCO datasets presented at major machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including recent state-of-the-art methods. All the source code, configurations, log files and trained model weights are publicly available at https://github.com/yoshitomo-matsubara/torchdistill .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑