从流中提取更多价值：为流式强化学习在线学习表示
Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
AI总结：
本文提出了一种改进的流式强化学习方法，通过自预测表示提升表示学习效率，有效解决采样效率低的问题，并在多个基准测试中表现出色。
摘要：
在流式强化学习（RL）中，转移在单次更新后立即被观察并丢弃。尽管这最大限度地减少了设备上的资源使用，但使智能体变得众所周知的采样效率低下，因为仅依靠基于价值的损失难以从瞬时数据中提取有意义的表示。我们提出将自预测表示（SPR）扩展到流式管道中，以最大限度地利用每个观察到的帧。然而，由于流式环境引起的高度相关样本，简单地应用此辅助损失会导致训练不稳定。因此，我们引入了与动量目标相对的正交梯度更新，并解决了由流式特定优化器引起的梯度冲突。在Atari、MinAtar和Octax套件上验证，我们的方法系统性地优于现有的流式基线。潜空间分析，包括t-SNE可视化和有效秩测量，证实我们的方法学习了更丰富的表示，弥合了由于没有回放缓冲区造成的性能差距，同时仍然足够高效，可以在几个CPU核心上进行训练。
英文摘要：
In streaming Reinforcement Learning (RL), transitions are observed and discarded immediately after a single update. While this minimizes resource usage for on-device applications, it makes agents notoriously sample-inefficient, since value-based losses alone struggle to extract meaningful representations from transient data. We propose extending Self-Predictive Representations (SPR) to the streaming pipeline to maximize the utility of every observed frame. However, due to the highly correlated samples induced by the streaming regime, naively applying this auxiliary loss results in training instabilities. Thus, we introduce orthogonal gradient updates relative to the momentum target and resolve gradient conflicts arising from streaming-specific optimizers. Validated across the Atari, MinAtar, and Octax suites, our approach systematically outperforms existing streaming baselines. Latent-space analysis, including t-SNE visualizations and effective-rank measurements, confirms that our method learns significantly richer representations, bridging the performance gap caused by the absence of a replay buffer, while remaining efficient enough to train on just a few CPU cores.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
