改进暗经验回放与水库采样以实现巩固与可塑性之间的更好平衡
Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity
AI总结：
本文改进暗经验回放与水库采样方法，通过自动适应权重、纠正错误数据和优化存储策略，实现持续学习中记忆巩固与可塑性的平衡。
摘要：
持续学习是自主代理最本质的能力之一，可以逐步学习日常生活技能。为了实现这一终极目标，一种简单但强大的方法，暗经验回放（DER），最近已被提出。DER通过随机存储流数据到水库采样（RS）缓冲区中，并通过重新学习它们或保留过去的输出来减轻灾难性遗忘，即过去获得的技能无意中被遗忘的现象。然而，由于DER考虑了多个目标，它在没有适当加权的情况下无法正常工作。此外，保留过去输出的能力会因分布偏移或其他因素导致过去输出不正确而抑制学习。这是巩固与可塑性之间的权衡。这种权衡甚至隐藏在RS缓冲区中，该缓冲区逐渐停止存储新技能的新数据，因为数据持续传递给它。为了缓解这种权衡并实现更好的平衡，本文提出了改进DER和RS的策略。具体而言，DER通过自动适应权重、阻止重新学习错误数据以及纠正过去输出进行改进。RS也通过接受概率的泛化、多缓冲区的分层以及有意省略不必要的数据进行改进。这些改进通过多个基准测试进行验证，包括回归、分类和强化学习问题。结果表明，所提出的方法通过平衡记忆巩固和可塑性，实现了学习性能的稳步提升。
英文摘要：
Continual learning is the one of the most essential abilities for autonomous agents, which can incrementally learn daily-life skills. For this ultimate goal, a simple but powerful method, dark experience replay (DER), has been proposed recently. DER mitigates catastrophic forgetting, in which the skills acquired in the past are unintentionally forgotten, by stochastically storing the streaming data in a reservoir sampling (RS) buffer and by relearning them or retaining the past outputs for them. However, since DER considers multiple objectives, it will not function properly without appropriate weighting of them. In addition, the ability to retain past outputs inhibits learning if the past outputs are incorrect due to distribution shift or other effects. This is due to a tradeoff between memory consolidation and plasticity. The tradeoff is hidden even in the RS buffer, which gradually stops storing new data for new skills in it as data is continuously passed to it. To alleviate the tradeoff and achieve better balance, this paper proposes improvement strategies to each of DER and RS. Specifically, DER is improved with automatic adaptation of weights, block of replaying erroneous data, and correction of past outputs. RS is also improved with generalization of acceptance probability, stratification of plural buffers, and intentional omission of unnecessary data. These improvements are verified through multiple benchmarks including regression, classification, and reinforcement learning problems. As a result, the proposed methods achieve steady improvements in learning performance by balancing the memory consolidation and plasticity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
