具有递推地 horizon 信用分配的行为约束强化学习
Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control
AI总结：
本文提出一种行为约束强化学习框架，通过递推地 horizon 预测机制和参考轨迹条件化策略，实现超越演示并保持专家行为一致的高性能控制策略。
摘要：
学习保持与专家行为一致的高性能控制策略是机器人领域的一个基本挑战。强化学习可以发现高性能策略，但往往偏离人类期望行为，而模仿学习受限于演示质量且难以超越专家数据。我们提出一种行为约束强化学习框架，能够在超越演示的同时显式控制偏离专家行为。由于动态控制中专家一致行为本质上是轨迹级的，我们引入递推地 horizon 预测机制，该机制建模短期未来轨迹并在训练中提供前瞻奖励。为了考虑人类行为在扰动和变化条件下的自然变异性，我们进一步将策略条件化于参考轨迹，使其能够表示专家一致行为的分布，而不是单一确定性目标。实证上，我们在高保真赛车模拟器中评估该方法，使用专业车手的数据，该领域以极端动态和狭窄性能差距为特征。所学策略在保持与专家驾驶行为紧密一致的同时实现了竞争性的圈速，优于基线方法在性能和模仿质量方面。除了标准基准外，我们还在驾驶者在环模拟器中进行人类基础评估，显示所学策略能再现依赖设置的驾驶特征，与顶级专业赛车手的反馈一致。这些结果表明，我们的方法能够学习既最优又行为一致的高性能控制策略，并可作为复杂控制系统中人类决策的可靠替代品。
英文摘要：
Learning high-performance control policies that remain consistent with expert behavior is a fundamental challenge in robotics. Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior, whereas imitation learning is limited by demonstration quality and struggles to improve beyond expert data. We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations while explicitly controlling deviation from expert behavior. Because expert-consistent behavior in dynamic control is inherently trajectory-level, we introduce a receding-horizon predictive mechanism that models short-term future trajectories and provides look-ahead rewards during training. To account for the natural variability of human behavior under disturbances and changing conditions, we further condition the policy on reference trajectories, allowing it to represent a distribution of expert-consistent behaviors rather than a single deterministic target. Empirically, we evaluate the approach in high-fidelity race car simulation using data from professional drivers, a domain characterized by extreme dynamics and narrow performance margins. The learned policies achieve competitive lap times while maintaining close alignment with expert driving behavior, outperforming baseline methods in both performance and imitation quality. Beyond standard benchmarks, we conduct human-grounded evaluation in a driver-in-the-loop simulator and show that the learned policies reproduce setup-dependent driving characteristics consistent with the feedback of top-class professional race drivers. These results demonstrate that our method enables learning high-performance control policies that are both optimal and behavior-consistent, and can serve as reliable surrogates for human decision-making in complex control systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
