Condensed datasets offer a compact representation of larger datasets, but training models directly on them or using them to enhance model performance through knowledge distillation (KD) can result in suboptimal outcomes due to limited information. To address this, we propose a method that expands condensed datasets using model inversion, a technique for generating synthetic data based on the impressions of a pre-trained model on its training data. This approach is particularly well-suited for KD scenarios, as the teacher model is already pre-trained and retains knowledge of the original training data. By creating synthetic data that complements the condensed samples, we enrich the training set and better approximate the underlying data distribution, leading to improvements in student model accuracy during knowledge distillation. Our method demonstrates significant gains in KD accuracy compared to using condensed datasets alone and outperforms standard model inversion-based KD methods by up to 11.4% across various datasets and model architectures. Importantly, it remains effective even when using as few as one condensed sample per class, and can also enhance performance in few-shot scenarios where only limited real data samples are available.
Contrastive Model Inversion for Data-Free Knowledge Distillation
Zhejiang University(浙江大学);Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies(浙江大学-阿里巴巴前沿技术联合研究中心);National University of Singapore(新加坡国立大学)
Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack
The University of Tokyo(东京大学);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
本文发现 FedMD 范式存在隐私泄露风险,提出 PLI 攻击,恶意服务器通过训练反演神经网络利用模型间的置信度差距,仅用配对 logits 即可高成功率重建私有图像。
Predictive Free Energy Simulations Through Hierarchical Distillation of Quantum Hamiltonians
Division of Chemistry and Chemical Engineering, California Institute of Technology, Pasadena, CA, 91125, USA(化学与化工系,加州理工学院,帕萨迪纳,CA,91125,USA);Chemical Engineering, California Institute of Technology, Pasadena, CA, 91125, USA(化学工程,加州理工学院,帕萨迪纳,CA,91125,USA)
Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
CFAR, Agency for Science, Technology and Research, Singapore(新加坡科技研究局应用科学与技术研究院);IHPC, Agency for Science, Technology and Research, Singapore(新加坡科技研究局信息与通信技术研究院);National University of Singapore(新加坡国立大学)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)