神经子空间重分配：基于检索的子空间内存管理的持续学习
Neural Subspace Reallocation: Continual Learning as Retrieval-Based Subspace Memory Management
AI总结：
提出神经子空间重分配（NSR），通过SVD压缩、相似性检索和重分配循环管理参数子空间内存，在循环环境中实现10倍恢复速度提升，并证明记忆机制比学习策略更关键。
摘要：
我们引入神经子空间重分配（NSR），它将持续学习重新定义为参数子空间上的内存管理。NSR不将低秩适配（LoRA）模块视为每个任务的即弃适配器，而是通过一个循环周期将其管理为冻结骨干网络上的可压缩、可检索的内存单元：（1）通过SVD压缩学到的LoRA，（2）将其保留在TaskKnowledgeBank中，（3）通过嵌入相似性召回相关的过去LoRA以热启动新任务或返回任务，以及（4）相应地重新分配活动子空间，并通过蒸馏保护先前任务。我们证明，在循环环境中，任何无内存的分配策略相对于由Bank支持的历史感知策略，都会产生累积遗憾Omega(T(M-1)Delta_switch)（定理1）。在Split-CIFAR-100上，Bank将循环恢复时间减少了10倍，与预测完全一致；在异构的5-Datasets基准上，NSR实现了最高准确率和最少遗忘，向后迁移比无内存启发式方法接近9倍更接近零。关键的是，我们进行了一项控制研究，隔离了哪个组件重要：固定Bank仅改变分配规则，我们发现简单的基于相似性的检索规则匹配或击败了学习的强化学习控制器（恢复重复任务分别需要0步和1.8步，并达到相同准确率）。因此，我们核心且诚实的发现是，在固定容量下，驱动持续学习性能的是记忆机制——压缩和相似性检索——而非学习的分配策略。内存预算分析证实，压缩后的Bank保持较小——每个任务0.29 MB的参数内存——因此top-K保留上限限制了总占用空间，同时为保留的任务保持快速恢复。
英文摘要：
We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks. We prove that in cyclic environments any memoryless allocation policy incurs cumulative regret Omega(T(M-1)Delta_switch) relative to a history-aware policy backed by the Bank (Theorem 1). Empirically, on Split-CIFAR-100 the Bank reduces cyclic recovery time by 10x, exactly as predicted, and on the heterogeneous 5-Datasets benchmark NSR achieves the highest accuracy and the least forgetting, about 9x closer to zero backward transfer than the memoryless heuristics. Crucially, we run a controlled study that isolates which component matters: holding the Bank fixed and varying only the allocation rule, we find that a simple similarity-based retrieval rule matches or beats a learned reinforcement-learning controller (recovering recurring tasks in 0 vs 1.8 steps and reaching equal accuracy). Our central, honest finding is therefore that the memory mechanism -- compression and similarity retrieval -- rather than a learned allocation policy, drives continual-learning performance under fixed capacity. A memory-budget analysis confirms the compressed Bank stays small -- 0.29 MB of parameter memory per task -- so a top-K retention cap bounds the total footprint while preserving fast recovery for retained tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
