arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.14538cs.CV
收藏

通过异构模型实现模型无关的数据集蒸馏

Towards Model-Agnostic Dataset Condensation by Heterogeneous Models

Kyung Hee University(庆熙大学)

Jun-Yeong Moon, Jung Uk Kim, Gyeong-Moon Park

AI总结:

针对数据集蒸馏图像特定于模型的问题,提出异构模型数据集蒸馏方法HMDC,通过梯度平衡模块和互蒸馏技术促进跨模型交互,生成通用蒸馏图像,提升实用性。

摘要:

深度学习的发展伴随着模型和可用数据的激增。数据集规模的激增及随之而来的计算需求激增促进了数据集蒸馏(DC)的发展。尽管先前的研究通过分布对齐和训练轨迹跟踪等方法生成合成图像以实现更高效的模型训练,但在实际使用这些蒸馏图像时出现了重大挑战。值得注意的是,这些蒸馏图像往往特定于特定模型,限制了其通用性和实用性。针对这一局限性,我们引入了一种新方法——异构模型数据集蒸馏(HMDC),旨在通过跨模型交互生成普遍适用的蒸馏图像。为解决使用异构模型时的梯度幅度差异和语义距离问题,我们提出了梯度平衡模块(GBM)和带有空间语义分解方法的互蒸馏(MD)。通过平衡各模型的贡献并使其语义保持紧密,我们的方法克服了模型特定蒸馏图像的限制,增强了更广泛的实用性。源代码可在 https://github.com/KHU-AGI/HMDC 获取。

英文摘要:

Abstract. The advancement of deep learning has coincided with the proliferation of both models and available data. The surge in dataset sizes and the subsequent surge in computational requirements have led to the development of the Dataset Condensation (DC). While prior studies have delved into generating synthetic images through methods like distribution alignment and training trajectory tracking for more efficient model training, a significant challenge arises when employing these condensed images practically. Notably, these condensed images tend to be specific to particular models, constraining their versatility and practicality. In response to this limitation, we introduce a novel method, Heterogeneous Model Dataset Condensation (HMDC), designed to produce universally applicable condensed images through cross-model interactions. To address the issues of gradient magnitude difference and semantic distance in models when utilizing heterogeneous models, we propose the Gradient Balance Module (GBM) and Mutual Distillation (MD) with the SpatialSemantic Decomposition method. By balancing the contribution of each model and maintaining their semantic meaning closely, our approach overcomes the limitations associated with model-specific condensed images and enhances the broader utility. The source code is available in https://github.com/KHU-AGI/HMDC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑