Humans learn continually throughout their lifespan by accumulating diverse knowledge and fine-tuning it for future tasks. When presented with a similar goal, neural networks suffer from catastrophic forgetting if data distributions across sequential tasks are not stationary over the course of learning. An effective approach to address such continual learning (CL) problems is to use hypernetworks which generate task dependent weights for a target network. However, the continual learning performance of existing hypernetwork based approaches are affected by the assumption of independence of the weights across the layers in order to maintain parameter efficiency. To address this limitation, we propose a novel approach that uses a dependency preserving hypernetwork to generate weights for the target network while also maintaining the parameter efficiency. We propose to use recurrent neural network (RNN) based hypernetwork that can generate layer weights efficiently while allowing for dependencies across them. In addition, we propose novel regularisation and network growth techniques for the RNN based hypernetwork to further improve the continual learning performance. To demonstrate the effectiveness of the proposed methods, we conducted experiments on several image classification continual learning tasks and settings. We found that the proposed methods based on the RNN hypernetworks outperformed the baselines in all these CL settings and tasks.
Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for Federated Continual Learning
National University of Singapore（新加坡国立大学）；Southeast University（东南大学）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Case Western Reserve University（凯斯西储大学）
Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCL
National University of Singapore（新加坡国立大学）；Southeast University（东南大学）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Case Western Reserve University（凯斯西储大学）
(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork
University of Exeter（埃克塞特大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of liverpool（利物浦大学）；Sun Yat-sen University（中山大学）；Hohai University（河海大学）；UNC-Chapel Hill（北卡罗来纳大学夏洛特分校）
EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning
Franklin College of Arts and Sciences（富兰克林文理学院）；University of Georgia（佐治亚大学）；Institute for Artificial Intelligence（人工智能研究所）；Mary Frances Early College of Education（玛丽·弗朗西斯·厄尔利教育学院）
SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense
Jagiellonian University, Faculty of Mathematics and Computer Science（杰洛内维大学数学与计算机科学学院）；Jagiellonian University, Doctoral School of Exact and Natural Sciences（杰洛内维大学精确与自然科学研究博士学院）；Akces NCBR；IDEAS Research Institute（IDEAS研究所）
Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
Chair of Data Science in Earth Observation, Technical University of Munich (TUM)（地球观测数据科学教授职位，慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；AIML Lab, School of Computer Science, University of St. Gallen（人工智能实验室，圣加尔登大学计算机科学学院）；School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens（农村、测绘与地理信息工程学院，国家技术大学雅典）；Image Processing Laboratory (IPL), Universitat de València（图像处理实验室（IPL），瓦伦西亚大学）
In Situ Training of Implicit Neural Compressors for Scientific Simulations via Sketch-Based Regularization
Applied Mathematics, University of Washington, Seattle（华盛顿大学应用数学系，西雅图）；Applied Mathematics, University of Colorado, Boulder（科罗拉多大学应用数学系，伯尔德）；Ann and H.J. Smead Department of Aerospace Engineering Sciences, University of Colorado, Boulder（科罗拉多大学伯尔德分校安与H.J. 塞梅德航空航天工程科学系）
