数据集蒸馏旨在将大型数据集压缩成一个显著更紧凑的合成数据集,同时不降低训练模型的性能。为了实现这一目标,现有方法使用代理模型从目标数据集中提取信息,并将其嵌入到蒸馏数据集中。因此,提取和嵌入信息的质量决定了蒸馏数据集的质量。在这项工作中,我们发现现有方法在信息提取和嵌入阶段都引入了错位信息。为了缓解这一问题,我们提出了Prioritize Alignment in Dataset Distillation (PAD),从以下两个视角对齐信息。1) 我们根据压缩比例对目标数据集进行剪枝,以过滤代理模型能够提取的信息。2) 我们仅使用代理模型的深层来执行蒸馏,以避免过度引入低级信息。这种简单策略有效过滤了错位信息,并为主流基于匹配的蒸馏算法带来了非平凡的改进。此外,基于轨迹匹配,PAD在多个基准上取得了显著改进,达到了最先进的性能。
英文摘要:
Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, \textbf{PAD} achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance.
DDTime: Dataset Distillation with Spectral Alignment and Information Bottleneck for Time-Series Forecasting
The City University of New York, CUNY(纽约城市大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Zhejiang University(浙江大学);University of Illinois at Chicago(伊利诺伊大学香槟分校);Tongji University(同济大学)
Omnimodal Dataset Distillation via High-order Proxy Alignment
University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);National University of Singapore(新加坡国立大学);The University of Sydney(悉尼大学)
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
Faculty of Information Technolody, Monash University, Australia(墨尔本大学信息科技学院,澳大利亚);School of Software, Northeastern University, China(东北大学软件学院,中国);Department of Medicine, National University of Singapore, Singapore(新加坡国立大学医学部,新加坡)
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China(电子工程学院,深圳研究生院,北京大学,深圳,中国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学,深圳,中国)
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
Pingtan Research Institute of Xiamen University(厦门大学滨海研究院);Imperial College London(伦敦帝国理工学院);University of Sussex(苏塞克斯大学);Purdue University(普渡大学);University of Liverpool(利物浦大学);JTB Technology Corp.(JTB技术公司);Emory University(埃默里大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);Kyoto University(京都大学);AppCubic;Georgia Institute of Technology(佐治亚理工学院);AI Agent Lab(AI代理实验室)
Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息与电子工程学院);ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院)
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education(大数据与人工智能交通运输联合实验室,(北京交通大学)教育部);School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(计算机科学与技术学院,北京交通大学,北京,中国);Tencent Inc, China(腾讯公司,中国)