While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design experiments by declarative PyYAML configuration files, and helps researchers complete the recently proposed ML Code Completeness Checklist. Using the developed framework, we demonstrate its various efficient training strategies, and implement a variety of knowledge distillation methods. We also reproduce some of their original experimental results on the ImageNet and COCO datasets presented at major machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including recent state-of-the-art methods. All the source code, configurations, log files and trained model weights are publicly available at https://github.com/yoshitomo-matsubara/torchdistill .
m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
University of Macau(澳门大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Hong Kong(香港大学);Beijing University of Posts and Telecommunications(北京邮电大学);Peking University(北京大学);HKUST(香港科技大学)
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
ETH Zurich(苏黎世联邦理工学院);Allen Institute for AI(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Knowledge distillation as a pathway toward next-generation intelligent ecohydrological modeling systems
Puget Sound Institute, University of Washington(华盛顿大学普吉特海湾研究所);Department of Civil Engineering, The University of Hong Kong(香港大学土木工程系);School for the Environment, University of Massachusetts Boston(马萨诸塞大学波士顿分校环境学院);Thornwell Labs(索恩韦尔实验室);Department of Hydrology and Atmospheric Sciences, The University of Arizona(亚利桑那大学水文与大气科学系)
Self-Distilled Vision Transformer for Domain Generalization
Mohamed Bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Oxford Brookes University(牛津布鲁克斯大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
Knowledge Fusion of Large Language Models Via Modular SkillPacks
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong Polytechnic University(香港理工大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学)