arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01301cs.LGcs.AIcs.NI
收藏

MARLIN：基于Soft Actor-Critic的强化学习用于真实网络中的拥塞控制

MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks

Florida Institute for Human & Machine Cognition (IHMC)（佛罗里达人机认知研究所（IHMC））；The University of West Florida (UWF)（西佛罗里达大学（UWF））；US Army Research Laboratory (ARL)（美国陆军研究实验室（ARL））

Raffaele Galliera, Alessandro Morelli, Roberto Fronteddu, Niranjan Suri

备注：
10 pages, 5 figures, AAAI 2023 workshop "Reinforcement Learning Ready for Production", accepted at NOMS 2023 - IEEE/IFIP Network Operations and Management Symposium
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对通用拥塞控制算法难以适应广泛场景的挑战，提出基于Soft Actor-Critic算法的强化学习框架MARLIN，通过在真实网络中训练以克服仿真到真实的差异，在文件传输任务中取得与TCP Cubic相当的性能，

摘要：

快速高效的传输协议是日益分布式的世界的基础。持续提供改进的通信性能以支持下一代应用和服务的负担，加上系统和网络技术日益增加的异构性，促进了在特定环境下表现良好的拥塞控制（CC）算法的设计。设计一种能适应广泛场景的通用CC算法仍是一个开放的研究问题。为应对这一挑战，我们提出应用一种新颖的强化学习（RL）方法。我们的解决方案MARLIN使用Soft Actor-Critic算法以最大化熵和回报，，并将学习过程建模为无限期任务。我们在具有变化背景流量模式的的真实网络上训练MARLIN，以克服研究人员在将RL应用于CC时遇到的仿真到真实的差异。我们在文件传输任务上评估了我们的解决方案，并将其与TCP Cubic进行了比较。虽然需要进一步研究，但结果表明表明，MARLIN在与其训练设置显著不同的任务中，只需很少的超参数调优微调，就能获得与TCP相当的结果。因此此，我们相信我们的工作代表了基于最大熵RL框架构建CC算法的有希望的第一步。

英文摘要：

Fast and efficient transport protocols are the foundation of an increasingly distributed world. The burden of continuously delivering improved communication performance to support next-generation applications and services, combined with the increasing heterogeneity of systems and network technologies, has promoted the design of Congestion Control (CC) algorithms that perform well under specific environments. The challenge of designing a generic CC algorithm that can adapt to a broad range of scenarios is still an open research question. To tackle this challenge, we propose to apply a novel Reinforcement Learning (RL) approach. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return and models the learning process as an infinite-horizon task. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch that researchers have encountered when applying RL to CC. We evaluated our solution on the task of file transfer and compared it to TCP Cubic. While further research is required, results have shown that MARLIN can achieve comparable results to TCP with little hyperparameter tuning, in a task significantly different from its training setting. Therefore, we believe that our work represents a promising first step toward building CC algorithms based on the maximum entropy RL framework.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑