论持续学习中宽度的收益递减
On the Diminishing Returns of Width for Continual Learning
AI总结：
本文提出持续学习理论分析框架，证明前馈网络中增加宽度虽可减轻灾难性遗忘，但收益会递减，并在更大宽度实验中验证该结论。
摘要：
尽管深度神经网络已在多种场景中展现出突破性性能，但当这些模型按顺序在新任务上训练时，往往会遭受灾难性遗忘。若干研究已通过实验证明，增加神经网络宽度会减轻灾难性遗忘，但尚未刻画宽度与持续学习之间的确切关系。我们设计了最早用于分析持续学习理论的框架之一，并证明在前馈网络（FFN）中，宽度与遗忘直接相关。具体而言，我们表明，通过增加网络宽度来减少遗忘会产生收益递减。我们在此前研究未曾探索的宽度上通过实验验证了这些主张，在这些宽度下可清楚观察到理论所预测的收益递减现象。
英文摘要：
While deep neural networks have demonstrated groundbreaking performance in various settings, these models often suffer from \emph{catastrophic forgetting} when trained on new tasks in sequence. Several works have empirically demonstrated that increasing the width of a neural network leads to a decrease in catastrophic forgetting but have yet to characterize the exact relationship between width and continual learning. We design one of the first frameworks to analyze Continual Learning Theory and prove that width is directly related to forgetting in Feed-Forward Networks (FFN). Specifically, we demonstrate that increasing network widths to reduce forgetting yields diminishing returns. We empirically verify our claims at widths hitherto unexplored in prior studies where the diminishing returns are clearly observed as predicted by our theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
