arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16939cs.LG
收藏

多任务与基于重放的持续学习中超参数化模型的理论洞察

Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning

University of Southern California(南加州大学)

Amin Banayeeanzade, Mahdi Soltanolkotabi, Mohammad Rostami

AI总结:

该研究以超参数化线性模型为代理,推导了多任务学习中模型规模等因素对性能的影响,以及基于重放的持续学习中缓冲区等对遗忘率的作用,并验证了结论适用于深度神经网络,为相关模型设计提供理论指导。

摘要:

多任务学习(Multi-task learning, MTL)是一种机器学习范式,旨在通过在多个相关任务上同时训练模型,提升其在这些任务上的泛化性能。与模型可即时获取所有任务训练数据的MTL不同,持续学习(continual learning, CL)需要随时间适应依次到来的新任务,同时不遗忘先前习得的知识。尽管CL和MTL在实践中应用广泛,且两个领域已有大量研究,但对于这类方法与深度神经网络等超参数化模型结合使用时的理论理解仍存在空白。本文将超参数化线性模型作为更复杂模型的代理展开研究。我们推导了描述MTL场景下各类系统参数对模型性能影响的理论结果,具体研究了模型规模、数据集规模和任务相似度对泛化误差与知识迁移的影响。此外,我们还提出了刻画基于重放的CL模型性能的理论结果。这些结果揭示了CL场景下缓冲区大小和模型容量对遗忘率的影响,有助于阐释部分当前最优的CL方法。最后,通过大量实证评估,我们证明了这些理论发现同样适用于深度神经网络,为实际设计MTL和CL模型提供了有价值的指导。

英文摘要:

Multi-task learning (MTL) is a machine learning paradigm that aims to improve the generalization performance of a model on multiple related tasks by training it simultaneously on those tasks. Unlike MTL, where the model has instant access to the training data of all tasks, continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theoretical understanding of these methods when used with overparameterized models such as deep neural networks. This paper studies the overparameterized linear models as a proxy for more complex models. We develop theoretical results describing the effect of various system parameters on the model's performance in an MTL setup. Specifically, we study the impact of model size, dataset size, and task similarity on the generalization error and knowledge transfer. Additionally, we present theoretical results to characterize the performance of replay-based CL models. Our results reveal the impact of buffer size and model capacity on the forgetting rate in a CL setup and help shed light on some of the state-of-the-art CL methods. Finally, through extensive empirical evaluations, we demonstrate that our theoretical findings are also applicable to deep neural networks, offering valuable guidance for designing MTL and CL models in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑