Plume：通过优先轨迹采样实现高性能深度强化学习网络控制器的框架
Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling
AI总结：
Plume框架通过三阶段优先轨迹采样，自动识别并平衡DRL训练数据中的偏斜分布，无需改动DRL流程，在多个网络环境中提升控制器性能，其ABR控制器Gelato在真实平台Puffer上显著减少卡顿达75%。
摘要：
深度强化学习（DRL）在各种网络环境中已展现出潜力。然而，这些环境给标准DRL技术带来了若干根本性挑战。它们难以探索，并表现出高水平的噪声和不确定性。尽管这些挑战使训练过程复杂化，但我们发现，在实践中，通过解决一个此前被忽视的因素——DRL训练数据集中偏斜的输入轨迹分布，我们可以大幅减轻其影响，甚至实现最先进的现实世界性能。我们引入了一个通用框架Plume，通过三阶段过程自动识别和平衡这种偏斜。首先，我们识别决定轨迹行为的关键特征。其次，我们将轨迹分类为簇。最后，我们对显著簇进行优先级排序，以提高控制器的整体性能。Plume可无缝地跨DRL算法工作，无需对DRL工作流程进行任何更改。我们在三个网络环境中评估了Plume，包括自适应比特率流媒体、拥塞控制和负载均衡。Plume在模拟和现实世界环境中，跨不同控制器和DRL算法均展现出优越性能。例如，我们新颖的ABR控制器Gelato，使用Plume训练，在直播平台Puffer上持续超过一年优于先前最先进的控制器。它是该平台上第一个在视频质量和卡顿方面均带来统计显著改进的控制器，将卡顿减少了高达75%。
英文摘要：
Deep Reinforcement Learning (DRL) has shown promise in various networking environments. However, these environments present several fundamental challenges for standard DRL techniques. They are difficult to explore and exhibit high levels of noise and uncertainty. Although these challenges complicate the training process, we find that in practice we can substantially mitigate their effects and even achieve state-of-the-art real-world performance by addressing a factor that has been previously overlooked: the skewed input trace distribution in DRL training datasets. We introduce a generalized framework, Plume, to automatically identify and balance the skew using a three-stage process. First, we identify the critical features that determine the behavior of the traces. Second, we classify the traces into clusters. Finally, we prioritize the salient clusters to improve the overall performance of the controller. Plume seamlessly works across DRL algorithms, without requiring any changes to the DRL workflow. We evaluated Plume on three networking environments, including Adaptive Bitrate Streaming, Congestion Control, and Load Balancing. Plume offers superior performance in both simulation and real-world settings, across different controllers and DRL algorithms. For example, our novel ABR controller, Gelato trained with Plume consistently outperforms prior state-of-the-art controllers on the live streaming platform Puffer for over a year. It is the first controller on the platform to deliver statistically significant improvements in both video quality and stalling, decreasing stalls by as much as 75%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
