Teddy：通过泰勒近似匹配实现高效的大规模数据集蒸馏
Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matching
AI总结：
本文提出Teddy框架，通过泰勒展开将多步梯度转化为一阶近似以降低内存消耗，并利用预缓存的弱模型池替代重复训练，在大规模数据集蒸馏上实现了高效且性能提升，在ImageNet上超越前人方法12.8%并减少46.6%运行时间。
摘要：
数据集蒸馏或压缩是指将大规模数据集压缩成一个更小的数据集，使得在该合成数据集上训练的模型能在真实数据上有效泛化。按定义解决这一挑战依赖于双层优化算法：在嵌套循环中每次迭代都训练一个新模型，并通过展开的计算图传播梯度。然而，该方法带来极高的内存和时间复杂度，难以扩展到ImageNet等大型数据集。为解决这些问题，本文引入了Teddy，一个旨在处理大规模数据集并提升效率的泰勒近似数据集蒸馏框架。一方面，在理论分析的支持下，我们提出一种源自泰勒展开的内存高效近似方法，将原本依赖多步梯度的形式转化为一阶形式。另一方面，我们发现在每次迭代中无需重复训练新模型，采用预缓存的弱模型池（可由单一基础模型生成），能同时提升时间效率和性能，特别是在处理大规模数据集时。大量实验表明，所提出的Teddy在Tiny-ImageNet和原始大小的ImageNet-1K数据集上达到了最先进的效率和性能，显著超越先前方法高达12.8%，同时减少了46.6%的运行时间。代码将发布于https://github.com/Lexie-YU/Teddy。
英文摘要：
Dataset distillation or condensation refers to compressing a large-scale dataset into a much smaller one, enabling models trained on this synthetic dataset to generalize effectively on real data. Tackling this challenge, as defined, relies on a bi-level optimization algorithm: a novel model is trained in each iteration within a nested loop, with gradients propagated through an unrolled computation graph. However, this approach incurs high memory and time complexity, posing difficulties in scaling up to large datasets such as ImageNet. Addressing these concerns, this paper introduces Teddy, a Taylor-approximated dataset distillation framework designed to handle large-scale dataset and enhance efficiency. On the one hand, backed up by theoretical analysis, we propose a memory-efficient approximation derived from Taylor expansion, which transforms the original form dependent on multi-step gradients to a first-order one. On the other hand, rather than repeatedly training a novel model in each iteration, we unveil that employing a pre-cached pool of weak models, which can be generated from a single base model, enhances both time efficiency and performance concurrently, particularly when dealing with large-scale datasets. Extensive experiments demonstrate that the proposed Teddy attains state-of-the-art efficiency and performance on the Tiny-ImageNet and original-sized ImageNet-1K dataset, notably surpassing prior methods by up to 12.8%, while reducing 46.6% runtime. Our code will be available at https://github.com/Lexie-YU/Teddy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
