arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.03360cs.LGcs.AI
收藏

在数据集蒸馏中优先考虑对齐

Prioritize Alignment in Dataset Distillation

National University of Singapore(新加坡国立大学);Carnegie Mellon University(卡内基梅隆大学);University of Toronto(多伦多大学);Shanghai AI Laboratory(上海人工智能实验室)

Zekai Li, Ziyao Guo, Wangbo Zhao, Tianle Zhang, Zhi-Qi Cheng, Samir Khaki, Kaipeng Zhang, Ahmad Sajedi, Konstantinos N Plataniotis, Kai Wang, Yang You

AI总结:

本文提出PAD方法,通过剪枝目标数据集并仅使用代理模型深层执行蒸馏,解决信息提取与嵌入阶段的错位问题,在多个基准上实现最先进性能。

摘要:

数据集蒸馏旨在将大型数据集压缩成一个显著更紧凑的合成数据集,同时不降低训练模型的性能。为了实现这一目标,现有方法使用代理模型从目标数据集中提取信息,并将其嵌入到蒸馏数据集中。因此,提取和嵌入信息的质量决定了蒸馏数据集的质量。在这项工作中,我们发现现有方法在信息提取和嵌入阶段都引入了错位信息。为了缓解这一问题,我们提出了Prioritize Alignment in Dataset Distillation (PAD),从以下两个视角对齐信息。1) 我们根据压缩比例对目标数据集进行剪枝,以过滤代理模型能够提取的信息。2) 我们仅使用代理模型的深层来执行蒸馏,以避免过度引入低级信息。这种简单策略有效过滤了错位信息,并为主流基于匹配的蒸馏算法带来了非平凡的改进。此外,基于轨迹匹配,PAD在多个基准上取得了显著改进,达到了最先进的性能。

英文摘要:

Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, \textbf{PAD} achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑