发表机构
Saxion University of Applied Sciences; University of Groningen; University of Twente（萨克逊应用科学大学; 格罗宁根大学; 特文特大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对强化学习在窄生存性任务中因探索提前终止而失败的问题，提出执行器动力学课程，通过退火关节刚度扩大生存性核，在Spot上成功实现四足到倒立过渡并迁移至硬件。
AI 中文摘要
强化学习已在广泛的腿式机器人任务中产生了有能力的控制器，但其中一部分任务在标准训练下无法收敛：这些任务中，大多数探索轨迹在产生有用的梯度信号之前就终止了。为了解决此类任务，我们引入了执行器动力学课程，这是一种将关节刚度初始化为高值，并随着完成回合长度的增加而将其退火至系统辨识值的程序。以小车-倒立摆系统作为代表性示例，我们表明在临界阻尼下更高的闭环关节固有频率会扩大底层马尔可夫决策过程的生存性核，从而增加任务可行的初始状态比例。我们在小车-倒立摆上验证了核的单调性，并将该课程应用于波士顿动力Spot上的四足到倒立过渡，这是一个窄生存性任务，在固定辨识刚度下训练会停滞在一个从未完成过渡的策略上。训练后的策略在10个随机种子下于仿真中执行该过渡，并成功迁移到硬件。更广泛地说，我们的结果表明，模拟执行器动力学是设计课程的一个有用维度，适用于探索受终止条件而非奖励信号瓶颈限制的任务。
英文摘要
Reinforcement learning has produced capable controllers across a broad range of legged-robot tasks, but a subset of these tasks fail to converge under standard training: those for which most exploration trajectories terminate before producing useful gradient signal. To address such tasks we introduce the \emph{Actuator Dynamics Curriculum}, a procedure that initializes joint stiffness at a high value and anneals it toward the system-identified value as completed episode lengths grow. Using a cart-pole system as a representative example, we show that higher closed-loop joint natural frequency under critical damping enlarges the viability kernel of the underlying Markov Decision Process, increasing the fraction of initial states from which the task is feasible. We validate the kernel monotonicity on the cart-pole and apply the curriculum to a quadrupedal-to-handstand transition on the Boston Dynamics Spot, a narrow-viability task where training under fixed identified stiffness plateaus at a policy that never completes the transition. The trained policy executes the transition in simulation across 10 seeds and transfers to hardware. More broadly, our results suggest that simulated actuator dynamics is a useful axis along which to design curricula for tasks in which exploration is bottlenecked by termination conditions rather than by reward signal.
CommentsAccepted at 2026 Conference on Robot Learning (CoRL)
