arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.08463cs.AIcs.LGcs.ROcs.SYeess.SY
收藏

利用强化学习探测反馈在技能习得中的作用

Privileged observations enable rapid and reliable policy discovery directly in the physical world

ETH Zürich(苏黎世联邦理工学院)

Antonio Terpin, Raffaello D'Andrea

AI总结:

本文利用强化学习研究反馈在技能习得中的作用,发现高维流反馈有助于发现高性能策略,但无反馈时仍能完成任务,表明学习条件取决于目标而非动态或策略复杂性。

摘要:

许多高水平的人类活动在几乎没有或没有外部反馈的情况下被执行:想想花样滑冰运动员完成三跳动作,投手投出带球旋转球以达到出局,或者咖啡师制作拉花。为了在完全受控的条件下研究技能习得过程,我们绕过了人类受试者。相反,我们直接将一个通用的强化学习代理与桌面上的旋转圆筒连接到循环水道中,以最大化或最小化阻力。该设置有几个有益的特性。首先,它是一个物理系统,具有丰富的相互作用和复杂的动态,只有物理世界才有:水流高度混沌且极难,如果不能的话,准确建模或模拟。其次,目标——最小化或最大化阻力——很容易说明,并且可以直接在奖励中捕捉,但好的策略在事先并不明显。第三,几十年来的实验研究提供了简单、高性能的开环策略的食谱。最后,该设置成本低廉,比人类研究更容易重复。在我们的实验中,我们发现高维流反馈使代理能够仅用几分钟的真实世界互动发现高性能的阻力控制策略。当我们后来回放相同的动作序列而没有任何反馈时,我们几乎获得相同的性能。这表明反馈,特别是流反馈,不是执行学习策略所必需的。令人惊讶的是,在训练过程中没有流反馈时,代理在阻力最大化方面无法发现任何有效的策略,但在阻力最小化方面仍然成功,但速度更慢且可靠性更低。我们的研究显示,学习高性能技能可能需要比执行更丰富的信息,学习条件可以是善良或恶劣的,这仅取决于目标,而不是动态或策略的复杂性。

英文摘要:

We study how privileged information about a physical system affects the discovery of high-performing policies when training a reinforcement learning agent directly in the physical world. We let the agent control a cylinder in a tabletop water channel to maximize or minimize drag. The flow is chaotic and difficult to model or simulate accurately and good strategies are not obvious beforehand. Decades-old experimental studies provide recipes for simple, high-performance, periodic open-loop policies that increase drag by 26.6% +/- 0.7% and reduce it by 29.7% +/- 1.3%. With dense observations of the cylinder wake, the agent learns within tens of minutes policies that increase drag by 25.5% +/- 0.9% and reduce it by 32.4% +/- 1.6%. We record action trajectories during online policy execution and replay them open loop as fixed sequences; these replays increase drag by 23.2% +/- 2.2% and reduce it by 32.1% +/- 3.0%. However, when we withhold flow observations during training, the agent still learns to decrease drag by 31.4% +/- 2.1%, but no run learns to increase it (1.8% +/- 4.3% drag increase). Our physical experiments demonstrate an extreme case: privileged observations can be decisive for policy discovery even when the resulting actions can be replayed in open loop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑