arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.20653cs.CVcs.AIcs.LG
收藏

数据集蒸馏与神经特征函数:一种极小极大视角

Dataset Distillation with Neural Characteristic Function: A Minmax Perspective

School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);Hong Kong University of Science and Technology, Guangzhou(香港科技大学(广州));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, Linfeng Zhang

AI总结:

本文提出基于神经特征函数差异的极小极大数据集蒸馏方法,通过优化特征函数采样和最小化分布差异,实现高效且高性能的数据蒸馏,显著提升准确率并大幅降低资源消耗。

摘要:

数据集蒸馏已成为深度学习中减少数据需求的一种强大方法。在各种方法中,基于分布匹配的方法因其计算效率与强大性能之间的平衡而脱颖而出。然而,现有分布匹配中使用的距离度量往往无法准确捕捉分布差异,导致差异度量不可靠。在本文中,我们将数据集蒸馏重新表述为一个极小极大优化问题,并引入神经特征函数差异(NCFD),这是一种全面且具有理论基础的度量,用于衡量分布差异。NCFD利用特征函数(CF)来封装完整的分布信息,采用神经网络优化CF频率参数的采样策略,从而最大化差异以增强距离估计。同时,我们在此优化的NCFD度量下最小化真实数据与合成数据之间的差异。我们的方法,称为神经特征函数匹配(\mymethod{}),在复平面上固有地对齐真实和合成数据的神经特征的相位和幅度,实现合成样本中真实性与多样性之间的平衡。实验表明,我们的方法在低分辨率和高分辨率数据集上均取得了显著的性能提升。值得注意的是,我们在ImageSquawk上实现了20.5%的准确率提升。与最先进的方法相比,我们的方法还将GPU内存使用量减少了300倍以上,并实现了20倍的处理速度提升。据我们所知,这是首次在单个NVIDIA 2080 Ti GPU上仅使用2.3 GB内存实现CIFAR-100的无损压缩的工作。

英文摘要:

Dataset distillation has emerged as a powerful approach for reducing data requirements in deep learning. Among various methods, distribution matching-based approaches stand out for their balance of computational efficiency and strong performance. However, existing distance metrics used in distribution matching often fail to accurately capture distributional differences, leading to unreliable measures of discrepancy. In this paper, we reformulate dataset distillation as a minmax optimization problem and introduce Neural Characteristic Function Discrepancy (NCFD), a comprehensive and theoretically grounded metric for measuring distributional differences. NCFD leverages the Characteristic Function (CF) to encapsulate full distributional information, employing a neural network to optimize the sampling strategy for the CF's frequency arguments, thereby maximizing the discrepancy to enhance distance estimation. Simultaneously, we minimize the difference between real and synthetic data under this optimized NCFD measure. Our approach, termed Neural Characteristic Function Matching (\mymethod{}), inherently aligns the phase and amplitude of neural features in the complex plane for both real and synthetic data, achieving a balance between realism and diversity in synthetic samples. Experiments demonstrate that our method achieves significant performance gains over state-of-the-art methods on both low- and high-resolution datasets. Notably, we achieve a 20.5\% accuracy boost on ImageSquawk. Our method also reduces GPU memory usage by over 300$\times$ and achieves 20$\times$ faster processing speeds compared to state-of-the-art methods. To the best of our knowledge, this is the first work to achieve lossless compression of CIFAR-100 on a single NVIDIA 2080 Ti GPU using only 2.3 GB of memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑