面向有监督在线持续学习的 Transformer
Transformers for Supervised Online Continual Learning
AI总结：
本文提出将 Transformer 的上下文条件机制、Transformer-XL 式在线 SGD 训练与回放结合，用于有监督在线持续学习，并在 CLOC 基准上显著超越现有最优结果。
摘要：
Transformer 已成为自然语言处理或音频处理等序列建模任务的主导架构，如今甚至被用于图像分类等并非天然具有序列性的任务。它们关注并处理一组 token 作为上下文的能力，使其能够发展出上下文少样本学习能力。然而，它们在在线持续学习中的潜力仍相对未被探索。在在线持续学习中，模型必须适应非平稳数据流，同时最小化累积的下一步预测损失。我们关注有监督在线持续学习设置，即为样本序列 $(x_t, y_t)$ 学习一个预测器 $x_t \ ightarrow y_t$。受 Transformer 的上下文学习能力及其与元学习联系的启发，我们提出一种利用这些优势进行在线持续学习的方法。我们的方法显式地让 Transformer 以近期观测为条件，同时按照 Transformer-XL 引入的流程，使用随机梯度下降对其进行在线训练。我们加入回放机制，以在遵守序列协议的同时保留多轮训练的优势。我们假设这种组合能够通过上下文学习实现快速适应，并通过参数化学习实现持续的长期改进。在 CLOC（一个具有挑战性的大规模图像地理定位真实世界基准）上，我们的方法相比此前最先进结果展现出显著提升。
英文摘要：
Transformers have become the dominant architecture for sequence modeling tasks such as natural language processing or audio processing, and they are now even considered for tasks that are not naturally sequential such as image classification. Their ability to attend to and to process a set of tokens as context enables them to develop in-context few-shot learning abilities. However, their potential for online continual learning remains relatively unexplored. In online continual learning, a model must adapt to a non-stationary stream of data, minimizing the cumulative nextstep prediction loss. We focus on the supervised online continual learning setting, where we learn a predictor $x_t \rightarrow y_t$ for a sequence of examples $(x_t, y_t)$. Inspired by the in-context learning capabilities of transformers and their connection to meta-learning, we propose a method that leverages these strengths for online continual learning. Our approach explicitly conditions a transformer on recent observations, while at the same time online training it with stochastic gradient descent, following the procedure introduced with Transformer-XL. We incorporate replay to maintain the benefits of multi-epoch training while adhering to the sequential protocol. We hypothesize that this combination enables fast adaptation through in-context learning and sustained longterm improvement via parametric learning. Our method demonstrates significant improvements over previous state-of-the-art results on CLOC, a challenging large-scale real-world benchmark for image geo-localization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
