arXiv 2505.133002025-09-21cs.CV

DD-Ranking：重新思考数据集蒸馏的评估

DD-Ranking: Rethinking the Evaluation of Dataset Distillation

NUS-HPC-AI-Lab（新加坡国立大学高性能计算与人工智能实验室）

Zekai Li, Xinhao Zhong, Samir Khaki, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Xuanlei Zhao, Wangbo Zhao, Ziheng Qin, Mengxuan Wu, Pengfei Zhou, Haon… 展开作者

Zekai Li, Xinhao Zhong, Samir Khaki, Zhiyuan Liang, Yuhao Zhou, Mingjia Shi, Ziqiao Wang, Xuanlei Zhao, Wangbo Zhao, Ziheng Qin, Mengxuan Wu, Pengfei Zhou, Haonan Wang, David Junhao Zhang, Jia-Wei Liu, Shaobo Wang, Dai Liu, Linfeng Zhang, Guang Li, Kun Wang, Zheng Zhu, Zhiheng Ma, Joey Tianyi Zhou, Jiancheng Lv, Yaochu Jin, Peihao Wang, Kaipeng Zhang, Lingjuan Lyu, Yiran Huang, Zeynep Akata, Zhiwei Deng, Xindi Wu, George Cazenavette, Yuzhang Shang, Justin Cui, Jindong Gu, Qian Zheng, Hao Ye, Shuo Wang, Xiaobo Wang, Yan Yan, Angela Yao, Mike Zheng Shou, Tianlong Chen, Hakan Bilen, Baharan Mirzasoleiman, Manolis Kellis, Konstantinos N. Plataniotis, Zhangyang Wang, Bo Zhao, Yang You, Kai Wang

备注：: 20 pages, 4 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对数据集蒸馏评估中准确率指标受额外技术干扰的问题，提出统一评估框架 DD-Ranking 及新指标，以更公平地衡量蒸馏数据集本身的信息增强效果。

摘要：

近年来，数据集蒸馏为数据压缩提供了一种可靠的解决方案，在生成的更小规模合成数据集上训练的模型能够取得与在原始数据集上训练的模型相当的性能。为了进一步提升合成数据集的性能，研究者提出了各种训练流程和优化目标，极大地推动了数据集蒸馏领域的发展。最近的解耦式数据集蒸馏方法在后评估阶段引入软标签和更强的数据增强，并将数据集蒸馏扩展到更大规模的数据集（如 ImageNet-1K）。然而，这引出一个问题：准确率是否仍然是公平评估数据集蒸馏方法的可靠指标？我们的实证研究发现，这些方法的性能提升往往来自额外技术，而非图像本身的内在质量，甚至随机采样的图像也能取得更优的结果。这种错位的评估设置严重阻碍了数据集蒸馏的发展。因此，我们提出 DD-Ranking，一个统一的评估框架，以及新的通用评估指标，以揭示不同方法所取得的真实性能提升。通过重新聚焦于蒸馏数据集的实际信息增强，DD-Ranking 为未来的研究进展提供了更全面、更公平的评估标准。

英文摘要：

In recent years, dataset distillation has provided a reliable solution for data compression, where models trained on the resulting smaller synthetic datasets achieve performance comparable to those trained on the original datasets. To further improve the performance of synthetic datasets, various training pipelines and optimization objectives have been proposed, greatly advancing the field of dataset distillation. Recent decoupled dataset distillation methods introduce soft labels and stronger data augmentation during the post-evaluation phase and scale dataset distillation up to larger datasets (e.g., ImageNet-1K). However, this raises a question: Is accuracy still a reliable metric to fairly evaluate dataset distillation methods? Our empirical findings suggest that the performance improvements of these methods often stem from additional techniques rather than the inherent quality of the images themselves, with even randomly sampled images achieving superior results. Such misaligned evaluation settings severely hinder the development of DD. Therefore, we propose DD-Ranking, a unified evaluation framework, along with new general evaluation metrics to uncover the true performance improvements achieved by different methods. By refocusing on the actual information enhancement of distilled datasets, DD-Ranking provides a more comprehensive and fair evaluation standard for future research advancements.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。