通过Koopman与控制解释强化学习模型行为
Interpreting Reinforcement Learning Model Behavior via Koopman with Control
AI总结:
本文通过Koopman与控制框架分析强化学习模型行为,发现其稳定性与可控性随训练变化,揭示了隐藏的进展度量。
摘要:
强化学习(RL)模型展示了学习复杂行为的能力,但定量评估这些行为对于安全性和发现新策略至关重要。通过将RL模型视为控制系统,我们假设数据驱动的Koopman算子近似可提供行为动态信息,从而增强可解释性。为此,我们应用Koopman与控制框架到多个标准基准环境训练的RL模型上,证明了拟合线性控制模型的稳定性与可控性在训练过程中以任务依赖的方式演变。跨不同训练时期或不同优化RL模型比较这些指标,有助于理解其差异。此外,我们发现即使RL模型获得的奖励保持不变,稳定性与可控性仍可能演变,预测进一步训练会增加奖励。这表明这些指标可能能作为隐藏的进展度量,这是机制可解释性中的核心思想。总体而言,我们的结果表明Koopman与控制框架提供了一种全面分析和解释RL模型行为,特别是在训练过程中的方式。
英文摘要:
Reinforcement learning (RL) models have shown the capability of learning complex behaviors, but quantitatively assessing those behaviors - which is critical for safety assurance and the discovery of novel strategies - is challenging. By viewing RL models as control systems, we hypothesize that data-driven approximations of their associated Koopman operators may provide dynamical information about their behavior, thus enabling greater interpretability. To test this, we apply the Koopman with control framework to RL models trained on several standard benchmark environments and demonstrate that properties of the fit linear control models, such as stability and controllability, evolve during training in a task dependent manner. Comparing these metrics across different training epochs or across differently optimized RL models enables an understanding of how they differ. In addition, we find cases where - even when the reward achieved by the RL model is static - the stability and controllability is nonetheless evolving, predicting increased reward with further training. This suggests that these metrics may be able to serve as hidden progress measures, a core idea in mechanistic interpretability. Taken together, our results illustrate that the Koopman with control framework provides a comprehensive way in which to analyze and interpret the behavior of RL models, particularly across training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。