arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.15089cs.LG
收藏

Where2Start:利用初始状态实现鲁棒且样本高效的强化学习

Where2Start: Leveraging initial States for Robust and Sample-Efficient Reinforcement Learning

Pouya Parsa, Raoof Zare Moayedi, Mohammad Bornosi, Mohammad Mahdi Bejani

AI总结:

提出Where2Start算法,通过选择使智能体附近状态更具不稳定性的初始状态来检索高信息量轨迹,从而显著提升强化学习的样本效率(高达8倍)与鲁棒性。

摘要:

关注如何计算梯度并选择下一个动作的强化学习算法,有效地提升了智能体的性能。然而,这些算法是与环境无关的。这意味着这些算法并未利用轨迹所捕获的知识。这导致算法必须采样大量轨迹来训练模型。通过考虑环境的本质以及智能体从该环境中的每个场景学到了多少,学习过程的策略可以被改变。该策略能检索到更具信息量的轨迹,因此智能体可以用更少的轨迹样本进行学习。我们提出了 Where2Start 算法,该算法选择初始状态,使得智能体在该状态附近具有更多的不稳定性。我们证明了这种选择方式减少了智能体达到可接受奖励所需采样的轨迹数量。我们的实验表明,Where2Start 可以将样本效率提高多达 8 倍。此外,Where2Start 可以与大多数最先进的算法相结合,并显著提升其鲁棒性和样本效率。

英文摘要:

The reinforcement learning algorithms that focus on how to compute the gradient and choose next actions, are effectively improved the performance of the agents. However, these algorithms are environment-agnostic. This means that the algorithms did not use the knowledge that has been captured by trajectory. This poses that the algorithms should sample many trajectories to train the model. By considering the essence of environment and how much the agent learn from each scenario in that environment, the strategy of the learning procedure can be changed. The strategy retrieves more informative trajectories, so the agent can learn with fewer trajectory sample. We propose Where2Start algorithm that selects the initial state so that the agent has more instability in vicinity of that state. We show that this kind of selection decreases number of trajectories that should be sampled that the agent reach to acceptable reward. Our experiments shows that Where2Start can improve sample efficiency up to 8 times. Also Where2Start can combined with most of state-of-the-art algorithms and improve that robustness and sample efficiency significantly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑