叠加中的超级掩码
Supermasks in Superposition
摘要：
我们提出了Supermasks in Superposition（SupSup）模型，能够连续学习数千个任务，而不会发生灾难性遗忘。我们的方法使用一个随机初始化的，固定的基础网络，并为每个任务找到一个子网络（超级掩码），实现良好的性能。如果在测试时给出任务标识，则可以使用最少的内存来检索正确的子网。如果没有提供，SupSup可以使用基于梯度的优化来推断任务，以找到最小化输出熵的学习超级掩码的线性叠加。在实践中，我们发现，一个单一的梯度步骤往往是足够的，以确定正确的掩模，即使在2500个任务。我们还展示了两个有前途的扩展。首先，SupSup模型可以完全在没有任务身份信息的情况下进行训练，因为它们可以检测到何时对新数据不确定，并为新的训练分布分配额外的超级掩码。最后，整个不断增长的超级掩码集可以通过隐式地将它们存储为固定大小的Hopfield网络中的吸引子来存储在恒定大小的水库中。
英文摘要：
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
