arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.11076cs.LGcs.AI
收藏

从MNIST到ImageNet再回到ImageNet:持续课程学习的基准

From MNIST to ImageNet and Back: Benchmarking Continual Curriculum Learning

AGH University of Science and Technology(AGH科技大学);University of Edinburgh(爱丁堡大学);American University(美利坚大学)

Kamil Faber, Dominik Zurek, Marcin Pietron, Nathalie Japkowicz, Antonio Vergari, Roberto Corizzo

摘要:

持续学习是近年来机器学习研究的热点之一。它的目标是超越机器学习中的经典假设,开发在动态环境中具有高鲁棒性的模型和学习策略。CL研究的景观被分割成几个学习评估协议,包括不同的学习任务,数据集和评估指标。此外,迄今为止采用的基准仍然远离现实世界场景的复杂性,并且通常是针对特定战略的特定能力而定制的。在这种情况下,很难客观地评估战略。在这项工作中,我们填补了这一空白CL图像数据引入两个新的CL基准,涉及多个异构的任务,从六个图像数据集,具有不同程度的复杂性和质量。我们的目标是公平地评估当前最先进的CL战略的共同点,更接近复杂的现实世界的情况。此外,我们还构建了我们的基准测试,以便根据课程以复杂性的递增和递减顺序呈现任务,以评估当前CL模型是否能够利用跨任务的结构。我们特别强调为CL社区提供一个严格的和可重复的评估协议,用于测量模型的泛化能力,并且在学习时不会忘记。此外,我们提供了一个广泛的实验评估表明,流行的CL策略,当挑战我们的基准,产生低于标准的性能,高水平的遗忘,并提出了一个有限的能力,有效地利用课程任务排序。我们认为,这些结果突出了需要在未来的CL作品进行严格的比较,以及铺平道路,设计新的CL策略,能够处理更复杂的情况。

英文摘要:

Continual learning (CL) is one of the most promising trends in recent machine learning research. Its goal is to go beyond classical assumptions in machine learning and develop models and learning strategies that present high robustness in dynamic environments. The landscape of CL research is fragmented into several learning evaluation protocols, comprising different learning tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted so far are still distant from the complexity of real-world scenarios, and are usually tailored to highlight capabilities specific to certain strategies. In such a landscape, it is hard to objectively assess strategies. In this work, we fill this gap for CL on image data by introducing two novel CL benchmarks that involve multiple heterogeneous tasks from six image datasets, with varying levels of complexity and quality. Our aim is to fairly evaluate current state-of-the-art CL strategies on a common ground that is closer to complex real-world scenarios. We additionally structure our benchmarks so that tasks are presented in increasing and decreasing order of complexity -- according to a curriculum -- in order to evaluate if current CL models are able to exploit structure across tasks. We devote particular emphasis to providing the CL community with a rigorous and reproducible evaluation protocol for measuring the ability of a model to generalize and not to forget while learning. Furthermore, we provide an extensive experimental evaluation showing that popular CL strategies, when challenged with our benchmarks, yield sub-par performance, high levels of forgetting, and present a limited ability to effectively leverage curriculum task ordering. We believe that these results highlight the need for rigorous comparisons in future CL works as well as pave the way to design new CL strategies that are able to deal with more complex scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑