防止在线持续自监督学习中潜在复习衰减的SOLAR
Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR
AI总结：
本文提出SOLAR方法，通过引入重叠损失和偏差代理，解决在线持续自监督学习中的潜在复习衰减问题，提升收敛速度和最终性能。
摘要：
本文探讨了在线持续自监督学习（OCSSL），即模型从连续的非平稳无标签数据流中学习，其中方法通常采用回放，快速收敛是关键。我们发现OCSSL需要特别关注稳定性-可塑性权衡：稳定方法（如回放与Reservoir采样）比可塑方法（如FIFO缓冲区）收敛更快，但在某些条件下会导致性能下降。我们通过潜在复习衰减假说解释这一崩溃现象，将其归因于过度稳定的回放导致的潜在空间退化。我们引入了两个指标（重叠和偏差）来诊断潜在退化，并与准确性下降相关联。基于这些见解，我们提出SOLAR，利用高效的在线偏差代理指导缓冲区管理，并结合显式重叠损失，使SOLAR能够自适应管理可塑性。实验表明，SOLAR在OCSSL视觉基准上实现了最先进的性能，具有高收敛速度和最终性能。
英文摘要：
This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
