Can pre-trained models assist in dataset distillation?
Zhejiang University of Technology(浙江工业大学);University of Electronic Science and Technology of China(电子科技大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学)
Yao Lu, Xuguang Chen, Yuchen Zhang, Jianyang Gu, Tianle Zhang, Yifan Zhang, Xiaoniu Yang, Qi Xuan, Kai Wang, Yang You
Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.
Sparse Depth-Guided Attention for Accurate Depth Completion: A Stereo-Assisted Monitored Distillation Approach
Huawei Cloud Computing Technology Co., Ltd.(华为云计算技术有限公司);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人学会);Institute of Robotics and Intelligent Manufacturing, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)机器人与智能制造研究院)
MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection
National Yang Ming Chiao Tung University(国立阳明交通大学);University of Washington(华盛顿大学);University of Southern California(南加州大学);Cisco Systems, Inc.(思科系统公司);Taijitu AI, Inc.(太极图人工智能公司);National Taiwan Normal University(国立台湾师范大学);National Taiwan University(国立台湾大学)
AMD: Automatic Multi-step Distillation of Large-scale Vision Models
University of Missouri – Kansas City(密苏里大学堪萨斯分校);Rochester Institute of Technology(罗切斯特理工学院);Meta AI;DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室);Santa Clara University(圣克拉拉大学);Kent State University(肯特州立大学);Virginia Tech(弗吉尼亚理工大学)
College of Computing and Informatics, Sungkyunkwan University(顺天大学计算与信息学院);Department of Computer Science and Engineering, University of Michigan(密歇根大学计算机科学与工程系);Konkuk Aerospace Design-Airworthiness Institute, Konkuk University(konkuk大学航空航天设计-适航研究所);Department of Electronic Systems, Aalborg University(奥尔堡大学电子系统系);Department of Information and Communication Engineering, Noakhali Science and Technology University(诺阿克利科学与技术大学信息与通信工程系)
Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition
Northwestern Polytechnical University(西北工业大学);Xijing Hospital, The Fourth Military Medical University(第四军医大学西京医院);Beijing Technology And Business University(北京工商大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nankai University(南开大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心)
Distilling a Powerful Student Model via Online Knowledge Distillation
Xiamen University(厦门大学);Tencent(腾讯);Pinterest;Peking University(北京大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus(中山大学深圳校区网络空间安全学院)
Prior Knowledge Distillation Network for Face Super-Resolution
School of Computer and Information, Hefei University of Technology(合肥工业大学计算机与信息学院);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院)
Self-similarity Prior Distillation for Unsupervised Remote Physiological Measurement
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);Artificial Intelligence Thrust, Hong Kong University of Science and Technology(香港科技大学(广州)人工智能学科)
HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors
Beihang University(北京航空航天大学);SenseTime Research(商汤科技研究院);ETH Zurich(苏黎世联邦理工学院);Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学)
A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI
Department of Computer Science and Engineering(计算机科学与工程系)
Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London(伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心(WEISS));The Chinese University of Hong Kong (CUHK)(香港中文大学(CUHK));Shun Hing Institute of Advanced Engineering, CUHK(香港中文大学信兴高等工程研究所)
Neural Tangent Knowledge Distillation for Optical Convolutional Networks
Department of Electrical and Computer Engineering, University of Washington(电气与计算机工程系,华盛顿大学);Department of Physics, University of Washington(物理系,华盛顿大学);Department of Applied Mathematics, University of Washington(应用数学系,华盛顿大学)
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
ZongmuTech(宗目科技);Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学脑智能科学与技术研究院);Huazhong University of Science and Technology(华中科技大学)
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
The Chinese University of Hong Kong(香港中文大学);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系);Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology(广东省计算机视觉与虚拟现实技术重点实验室);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)