arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.04838cs.CVcs.LG
收藏

超越随机:数据集蒸馏中的自动内循环优化

Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation

University of Electronic Science and Technology of China(电子科技大学)

Muquan Li, Hang Gou, Dongyang Zhang, Shuang Liang, Xiurui Xie, Deqiang Ouyang, Ke Qin

AI总结:

本文提出AT-BPTT框架,通过动态调整截断位置和窗口大小,提升数据集蒸馏的内循环优化性能,实现准确率提升和计算效率提升。

摘要:

随着对高效深度学习需求的增长,数据集蒸馏已成为一种关键技术,用于在压缩训练数据集的同时保持模型性能。然而,现有数据集蒸馏的内循环优化方法通常依赖于随机截断策略,这些策略缺乏灵活性,往往产生次优结果。在本工作中,我们观察到神经网络在不同的训练阶段——早期、中期和晚期——表现出不同的学习动态,使随机截断无效。为了解决这一限制,我们提出了自动截断回传时间(AT-BPTT)这一新框架,该框架根据内在梯度行为动态调整截断位置和窗口大小。AT-BPTT引入了三个关键组件:(1)一个概率机制用于阶段感知的时间步选择,(2)基于梯度变化的自适应窗口大小策略,以及(3)低秩Hessian近似以减少计算开销。在CIFAR-10、CIFAR-100、Tiny-ImageNet和ImageNet-1K上的广泛实验表明,AT-BPTT实现了最先进的性能,比基线方法平均提高了6.16%的准确率。此外,我们的方法通过3.9倍加速内循环优化,同时节省了63%的内存成本。

英文摘要:

The growing demand for efficient deep learning has positioned dataset distillation as a pivotal technique for compressing training dataset while preserving model performance. However, existing inner-loop optimization methods for dataset distillation typically rely on random truncation strategies, which lack flexibility and often yield suboptimal results. In this work, we observe that neural networks exhibit distinct learning dynamics across different training stages-early, middle, and late-making random truncation ineffective. To address this limitation, we propose Automatic Truncated Backpropagation Through Time (AT-BPTT), a novel framework that dynamically adapts both truncation positions and window sizes according to intrinsic gradient behavior. AT-BPTT introduces three key components: (1) a probabilistic mechanism for stage-aware timestep selection, (2) an adaptive window sizing strategy based on gradient variation, and (3) a low-rank Hessian approximation to reduce computational overhead. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show that AT-BPTT achieves state-of-the-art performance, improving accuracy by an average of 6.16% over baseline methods. Moreover, our approach accelerates inner-loop optimization by 3.9x while saving 63% memory cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑