Deep learning has grown tremendously over recent years, yielding state-of-the-art results in various fields. However, training such models requires huge amounts of data, increasing the computational time and cost. To address this, dataset distillation was proposed to compress a large training dataset into a smaller synthetic one that retains its performance -- this is usually done by (1) uniformly initializing a synthetic set and (2) iteratively updating/learning this set according to a predefined loss by uniformly sampling instances from the full data. In this paper, we improve both phases of dataset distillation: (1) we present a provable, sampling-based approach for initializing the distilled set by identifying important and removing redundant points in the data, and (2) we further merge the idea of data subset selection with dataset distillation, by training the distilled set on ``important'' sampled points during the training procedure instead of randomly sampling the next batch. To do so, we define the notion of importance based on the relative contribution of instances with respect to two different loss functions, i.e., one for the initialization phase (a kernel fitting function for kernel ridge regression and $K$-means based loss function for any other distillation method), and the relative cross-entropy loss (or any other predefined loss) function for the training phase. Finally, we provide experimental results showing how our method can latch on to existing dataset distillation techniques and improve their performance.
University of Chicago（芝加哥大学）；University of Southern California（南加州大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessian sketch is an optimization process with preconditioning, and develop accelerated iterative Hessian sketch via the searching the conjugate direction; we then establish primal-dual connections between the Hessian sketch and dual random projection, and apply the preconditioned conj…
Category Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
IDEAS NCBR；Warsaw University of Technology（华沙理工大学）；Gdańsk University of Technology（格但斯克理工大学）；Tooploox；Autonomous University of Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Attributes of sound inherent to objects can provide valuable cues to learn rich representations for object detection and tracking. Furthermore, the co-occurrence of audiovisual events in videos can be exploited to localize objects over the image field by solely monitoring the sound in the environment. Thus far, this has only been feasible in scenarios where the camera is static and for single object detection. Moreover, the robustness of these methods has been limited as they primarily rely on RGB images which are highly susceptible to illumination and weather changes. In this work, we present…
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Peking University（北京大学）；National Engineering Research Center of Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（高可信软件技术教育部重点实验室）；Center on Frontiers of Computing Studies, Peking University（北京大学计算前沿研究中心）
When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机系）
Large Language Model Meets Graph Neural Network in Knowledge Distillation
School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）；School of Computer Science and Technology, Donghua University（东华大学计算机科学与技术学院）；School of Computer and Information Engineering, Shanghai Polytechnic University（上海第二工业大学计算机与信息工程学院）；Department of Computer Science and Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校计算机科学与工程系）
Pusan National University（釜山国立大学）；University of Technology Sydney（悉尼科技大学）；University of Warwick（华威大学）
In real-world applications, Federated Learning (FL) meets two challenges: (1) scalability, especially when applied to massive IoT networks; and (2) how to be robust against an environment with heterogeneous data. Realizing the first problem, we aim to design a novel FL framework named Full-stack FL (F2L). More specifically, F2L utilizes a hierarchical network architecture, making extending the FL network accessible without reconstructing the whole network system. Moreover, leveraging the advantages of hierarchical network design, we propose a new label-driven knowledge distillation (LKD) techn…
Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions
School of Electronics and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Aerospace Communication and Networking Technology, Harbin Institute of Technology（哈尔滨工业大学广东省天地一体化信息网络重点实验室）
Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning
School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）；The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；No.1 Orthopedic Hospital of Chengdu（成都市第一骨科医院）
Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Telecommunications Engineering, Xidian University（西安电子科技大学电信工程学院）
Princeton University（普林斯顿大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Membership inference attacks are a key measure to evaluate privacy leakage in machine learning (ML) models. These attacks aim to distinguish training members from non-members by exploiting differential behavior of the models on member and non-member inputs. The goal of this work is to train ML models that have high membership privacy while largely preserving their utility; we therefore aim for an empirical membership privacy guarantee as opposed to the provable privacy guarantees provided by techniques like differential privacy, as such techniques are shown to deteriorate model utility. Specif…
Constrained Edge AI Deployment: Fine-Tuning vs Distillation for LLM Compression
Department of Computer Science, University of West Florida（计算机科学系，西佛罗里达大学）；Department of Mathematics and Statistics, University of West Florida（数学与统计学系，西佛罗里达大学）；Department of Intelligent Systems and Robotics, University of West Florida（智能系统与机器人系，西佛罗里达大学）；Institute of Human and Machine Cognition（人机认知研究所）；DEVCOM Army Research Laboratory（陆军研究实验室）
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
University of Utah（犹他大学）；University of Waterloo（滑铁卢大学）；The University of Queensland（昆士兰大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Virginia（弗吉尼亚大学）
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）；Vanderbilt University（范德堡大学）；Mayo Clinic Arizona（亚利桑那梅奥诊所）；Augusta University（奥古斯塔大学）
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
With the increasing popularity of deep learning on edge devices, compressing large neural networks to meet the hardware requirements of resource-constrained devices became a significant research direction. Numerous compression methodologies are currently being used to reduce the memory sizes and energy consumption of neural networks. Knowledge distillation (KD) is among such methodologies and it functions by using data samples to transfer the knowledge captured by a large model (teacher) to a smaller one(student). However, due to various reasons, the original training data might not be accessi…
The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitab…
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg（卢森堡大学安全、可靠性与信任跨学科中心）；Department of Computer Science, Peking University（北京大学计算机科学系）；Institute of Space Internet, Fudan University（复旦大学空间互联网研究院）；Department of Electrical and Computer Engineering, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Engineering, Edith Cowan University（埃迪斯科文大学工程学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
