Dataset Distillation with Neural Characteristic Function: A Minmax Perspective
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);Hong Kong University of Science and Technology, Guangzhou(香港科技大学(广州));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
数据集蒸馏已成为深度学习中减少数据需求的一种强大方法。在各种方法中,基于分布匹配的方法因其计算效率与强大性能之间的平衡而脱颖而出。然而,现有分布匹配中使用的距离度量往往无法准确捕捉分布差异,导致差异度量不可靠。在本文中,我们将数据集蒸馏重新表述为一个极小极大优化问题,并引入神经特征函数差异(NCFD),这是一种全面且具有理论基础的度量,用于衡量分布差异。NCFD利用特征函数(CF)来封装完整的分布信息,采用神经网络优化CF频率参数的采样策略,从而最大化差异以增强距离估计。同时,我们在此优化的NCFD度量下最小化真实数据与合成数据之间的差异。我们的方法,称为神经特征函数匹配(\mymethod{}),在复平面上固有地对齐真实和合成数据的神经特征的相位和幅度,实现合成样本中真实性与多样性之间的平衡。实验表明,我们的方法在低分辨率和高分辨率数据集上均取得了显著的性能提升。值得注意的是,我们在ImageSquawk上实现了20.5%的准确率提升。与最先进的方法相比,我们的方法还将GPU内存使用量减少了300倍以上,并实现了20倍的处理速度提升。据我们所知,这是首次在单个NVIDIA 2080 Ti GPU上仅使用2.3 GB内存实现CIFAR-100的无损压缩的工作。
英文摘要:
Dataset distillation has emerged as a powerful approach for reducing data requirements in deep learning. Among various methods, distribution matching-based approaches stand out for their balance of computational efficiency and strong performance. However, existing distance metrics used in distribution matching often fail to accurately capture distributional differences, leading to unreliable measures of discrepancy. In this paper, we reformulate dataset distillation as a minmax optimization problem and introduce Neural Characteristic Function Discrepancy (NCFD), a comprehensive and theoretically grounded metric for measuring distributional differences. NCFD leverages the Characteristic Function (CF) to encapsulate full distributional information, employing a neural network to optimize the sampling strategy for the CF's frequency arguments, thereby maximizing the discrepancy to enhance distance estimation. Simultaneously, we minimize the difference between real and synthetic data under this optimized NCFD measure. Our approach, termed Neural Characteristic Function Matching (\mymethod{}), inherently aligns the phase and amplitude of neural features in the complex plane for both real and synthetic data, achieving a balance between realism and diversity in synthetic samples. Experiments demonstrate that our method achieves significant performance gains over state-of-the-art methods on both low- and high-resolution datasets. Notably, we achieve a 20.5\% accuracy boost on ImageSquawk. Our method also reduces GPU memory usage by over 300$\times$ and achieves 20$\times$ faster processing speeds compared to state-of-the-art methods. To the best of our knowledge, this is the first work to achieve lossless compression of CIFAR-100 on a single NVIDIA 2080 Ti GPU using only 2.3 GB of memory.
Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(数据科学与人工智能系,香港理工大学);Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);The Hong Kong Polytechnic University Shenzhen Research Institute(香港理工大学深圳研究院)
Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective
University of Chinese Academy of Sciences(中国科学院大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Electronic Science and Technology of China(电子科技大学);Tsinghua University(清华大学)
Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息与电子工程学院);ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院)
Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
Hong Kong University of Science and Technology(香港科技大学)
当识别情绪时,情绪表现中的细微差别会产生情绪感知的模糊性或不确定性。情感不确定性以前被解释为多个注释者之间的评分者之间的分歧。在本文中,我们考虑了一个更常见和更具挑战性的场景:当只有单个情感标签可用时,对情感不确定性进行建模。从贝叶斯的角度来看,我们建议使用深度集成来捕获多个情感描述符的不确定性,即,动作单元、离散表达式标签和连续描述符。我们进一步应用迭代自蒸馏。多代迭代蒸馏显著提高了情感识别和不确定性估计的性能。我们的方法生成单个学生模型,该模型为域内样本提供准确的不确定性估计,并且生成可以检测域外样本的学生集合。我们使用Aff-wild 2数据集进行的情感识别和不确定性估计的实验表明,我们的算法比Temperature Scaling和Monte Carol Dropout给出了更可靠的不确定性估计。
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University(江苏大学);Griffith University(格里菲斯大学);National University of Singapore(新加坡国立大学);University of New South Wales(新南威尔士大学);Fujian University of Technology(福建理工大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);German Research Center for Artificial Intelligence(德国人工智能研究中心);Tsinghua University(清华大学);Technical University of Munich(慕尼黑工业大学);Imperial College London(伦敦帝国学院);Queen Mary University of London(伦敦玛丽女王大学)
Knowledge Distillation via Route Constrained Optimization
SenseTime Research(商汤科技研究院);National University of Defense Technology(国防科技大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学);Tsinghua University(清华大学)
Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所人机混合增强智能技术国家重点实验室);Guangdong Institute of Intelligence Science and Technology(广东省智能科学与技术研究院);Institute of Collaborative Innovation, University of Macau(澳门大学协同创新研究院);Beijing Huahang Institute of Radio Measurement(北京华航无线电测量研究所)
Omnimodal Dataset Distillation via High-order Proxy Alignment
University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);National University of Singapore(新加坡国立大学);The University of Sydney(悉尼大学)