通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2505.20970cs.LG
持续学习中表征偏移的度量：线性变换视角
Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective
Department of Electrical \& Electronic Engineering, Yonsei University, Seoul, South Korea；Department of Statistics；Data Science, Yonsei University, Seoul, South Korea
In continual learning scenarios, catastrophic forgetting of previously learned tasks is a critical issue, making it essential to effectively measure such forgetting. Recently, there has been growing interest in focusing on representation forgetting, the forgetting measured at the hidden layer. In this paper, we provide the first theoretical analysis of representation forgetting and use this analysis to better understand the behavior of continual learning. First, we introduce a new metric called representation discrepancy, which measures the difference between representation spaces constructed by two snapshots of a model trained through continual learning. We demonstrate that our proposed metric serves as an effective surrogate for the representation forgetting while remaining analytically tractable. Second, through mathematical analysis of our metric, we derive several key findings about the dynamics of representation forgetting: the forgetting occurs more rapidly to a higher degree as the layer index increases, while increasing the width of the network slows down the forgetting process. Third, we support our theoretical findings through experiments on real image datasets, including Split-CIFAR100 and ImageNet1K.
Stanford University（斯坦福大学）；New York University（纽约大学）
We consider the continual representation learning setting: sequentially pretrain a model $M'$ on tasks $T_1, \ldots, T_T$, and then adapt $M'$ on a small amount of data from each task $T_i$ to check if it has forgotten information from old tasks. Under a kNN adaptation protocol, prior work shows that continual learning methods improve forgetting over naive training (SGD). In reality, practitioners do not use kNN classifiers -- they use the adaptation method that works best (e.g., fine-tuning) -- here, we find that strong continual learning baselines do worse than naive training. Interestingly,…
New York University（纽约大学）；KAIST（韩国科学技术院）；Microsoft Research（微软研究院）；AITRICS
Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the le…
Humans learn adaptively and efficiently throughout their lives. However, incrementally learning tasks causes artificial neural networks to overwrite relevant information learned about older tasks, resulting in 'Catastrophic Forgetting'. Efforts to overcome this phenomenon often utilize resources poorly, for instance, by growing the network architecture or needing to save parametric importance scores, or violate data privacy between tasks. To tackle this, we propose SPACE, an algorithm that enables a network to learn continually and efficiently by partitioning the learnt space into a Core space…
SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；The 63rd Research Institute, National University of Defense Technology, Nanjing（国防科技大学第六十三研究所，南京）
Catastrophic forgetting in neural networks is a significant problem for continual learning. A majority of the current methods replay previous data during training, which violates the constraints of an ideal continual learning system. Additionally, current approaches that deal with forgetting ignore the problem of catastrophic remembering, i.e. the worsening ability to discriminate between data from different tasks. In our work, we introduce Relevance Mapping Networks (RMNs) which are inspired by the Optimal Overlap Hypothesis. The mappings reflects the relevance of the weights for the task at …
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University（东北大学克劳尔计算机科学学院）；New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）
Peking University（北京大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
It has been observed that neural networks perform poorly when the data or tasks are presented sequentially. Unlike humans, neural networks suffer greatly from catastrophic forgetting, making it impossible to perform life-long learning. To address this issue, memory-based continual learning has been actively studied and stands out as one of the best-performing methods. We examine memory-based continual learning and identify that large variation in the representation space is crucial for avoiding catastrophic forgetting. Motivated by this, we propose to diversify representations by using two typ…
ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
Key Laboratory of High Confidence Software Technologies, Ministry of Education（高可信软件技术重点实验室，教育部）；School of Computer Science, Peking University（北京大学计算机学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Repurposing Adversarial Perturbations for Continual Learning: From Defense to Active Alignment
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Deakin University（德肯大学）；Harbin Engineering University（哈尔滨工程大学）
Conformal Uncertainty Indicator for Continual Test-Time Adaptation
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所新模式识别实验室）；China Nuclear Power Engineering Co., LTD（中国核电工程有限公司）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；School of Computer Science & Engineering, LinYi University（临沂大学计算机科学与工程学院）；School of Electric & Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）
University of Tübingen（蒂宾根大学）；Yale University（耶鲁大学）
We propose a compression based continual task learning method that can dynamically grow a neural network. Inspired from the recent model compression techniques, we employ compression-aware training and perform low-rank weight approximations using singular value decomposition (SVD) to achieve network compaction. By encouraging the network to learn low-rank weight filters, our method achieves compressed representations with minimal performance degradation without the need for costly fine-tuning. Specifically, we decompose the weight filters using SVD and train the network on incremental tasks in…
While neural networks are powerful function approximators, they suffer from catastrophic forgetting when the data distribution is not stationary. One particular formalism that studies learning under non-stationary distribution is provided by continual learning, where the non-stationarity is imposed by a sequence of distinct tasks. Most methods in this space assume, however, the knowledge of task boundaries, and focus on alleviating catastrophic forgetting. In this work, we depart from this view and move the focus towards faster remembering -- i.e measuring how quickly the network recovers perf…
Sequoia: A Software Framework to Unify Continual Learning Research
ElementAI, a ServiceNow Company（ElementAI（ServiceNow旗下公司））；Mila - Quebec AI Institute（Mila - 魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；HEC Montréal（蒙特利尔高等商学院）；IBM Research（IBM研究院）
New York University（纽约大学）；DeepMind（深度思维）；Google Research（谷歌研究院）
Learning new tasks continuously without forgetting on a constantly changing data distribution is essential for real-world problems but extremely challenging for modern deep learning. In this work we propose HCL, a Hybrid generative-discriminative approach to Continual Learning for classification. We model the distribution of each task and each class with a normalizing flow. The flow is used to learn the data distribution, perform classification, identify task changes, and avoid forgetting, all leveraging the invertibility and exact likelihood which are uniquely enabled by the normalizing flow …
Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning
China University of Petroleum（中国石油大学）；WeBank（微众银行）；Pazhou Laboratory（琶洲实验室）；South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
FlyPrompt: Brain-Inspired Random-Expanded Routing with Temporal-Ensemble Experts for General Continual Learning
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University（生命科学学院，IDG/麦克格雷文脑科学研究院，清华大学）；Department of Psychological and Cognitive Sciences, Tsinghua University（心理学与认知科学系，清华大学）；School of Medicine, Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区医学学院）
Continual Learning Should Move Beyond Incremental Classification
TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI（黑森人工智能中心）；Polytechnic University of Turin（都灵理工大学）；Italian Institute of Technology（意大利技术研究院）；University of Pisa（比萨大学）；University of Cambridge（剑桥大学）；Bielefeld University（比勒费尔德大学）；University of Bremen（不来梅大学）；KU Leuven（鲁汶大学）；German Research Center for AI（德国人工智能研究中心）
ServiceNow（ServiceNow公司）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Heuritech（Heuritech公司）；HEC Montréal（蒙特利尔高等商学院）；Sorbonne University（索邦大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能讲席）
Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains. Motivated by this, we study the efficacy of pre-trained vision models as a foundation for downstream continual learning (CL) scenarios. Our goal is twofold. First, we want to understand the compute-accuracy trade-off between CL in the raw-data space and in the latent space of pre-trained encoders. Second, we investigate how the characteristics of the encoder, the pre-training algorithm and data, as well as of the resulting la…
Self-paced Weight Consolidation for Continual Learning
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Quantifying the information content in a neural network model is essentially estimating the model's Kolmogorov complexity. Recent success of prequential coding on neural networks points to a promising path of deriving an efficient description length of a model. We propose a practical measure of the generalizable information in a neural network model based on prequential coding, which we term Information Transfer ($L_{IT}$). Theoretically, $L_{IT}$ is an estimation of the generalizable part of a model's information content. In experiments, we show that $L_{IT}$ is consistently correlated with g…
Similarity of Neural Network Models: A Survey of Functional and Representational Measures
University of Passau（帕绍大学）；University of Mannheim（曼海姆大学）；RWTH Aachen University（亚琛工业大学）；GESIS - Leibniz Institute for the Social Sciences（GESIS莱布尼茨社会科学研究所）；Complexity Science Hub Vienna（维也纳复杂性科学中心）
University of Maryland, College Park（马里兰大学帕克分校）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；PUCRS（巴西圣保罗天主教大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Cambridge（剑桥大学）
A major challenge in studying robustness in deep learning is defining the set of ``meaningless'' perturbations to which a given Neural Network (NN) should be invariant. Most work on robustness implicitly uses a human as the reference model to define such perturbations. Our work offers a new view on robustness by using another reference NN to define the set of perturbations a given NN should be invariant to, thus generalizing the reliance on a reference ``human NN'' to any NN. This makes measuring robustness equivalent to measuring the extent to which two NNs share invariances, for which we pro…
NEC Corporation（日本电气股份有限公司）；Stanford University（斯坦福大学）；UC San Diego（加州大学圣迭戈分校）
Today's robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data, such as rare scenes of road construction sites, to steadily improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central compute servers (or the "cloud") places an enormous time and cost burden on network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system …
Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
Department of Computer Science and Engineering, UC San Diego（计算机科学与工程系，UC San Diego）；Department of Electrical and Computer Engineering, UC San Diego（电气与计算机工程系，UC San Diego）；Department of Cognitive Science, UC San Diego（认知科学系，UC San Diego）
National Technical University of Athens（国家技术大学雅典）；Archimedes RU, Athena RC（阿基米德RU，雅典RC）；Institute of Language and Speech Processing, Athena RC（语言与语音处理研究所，雅典RC）
