ACTER：用于解释和诊断强化学习策略的多样且可操作的逆事实序列
ACTER: Diverse and Actionable Counterfactual Sequences for Explaining and Diagnosing RL Policies
AI总结：
ACTER提出一种算法，利用NSGA-II进化算法生成多样且可操作的逆事实动作序列，以解释和诊断强化学习策略中的失败，并在两个环境中验证其有效性。
摘要：
理解强化学习（RL）中失败是如何发生的以及如何预防失败，对于实现调试、维护用户信任和开发个性化策略至关重要。逆事实推理常被用于通过搜索失败得以避免的最接近可能世界来归因责任和理解失败。然而，当前RL中的逆事实状态解释仅能利用当前状态特征解释结果，并且对于如何预防负面结果不提供可操作的补救措施。在这项工作中，我们提出了ACTER（用于解释强化学习结果的可操作逆事实序列），一种生成逆事实序列的算法，该算法提供关于如何避免失败的可操作建议。ACTER调查导致失败的动作，并使用进化算法NSGA-II生成逆事实动作序列，这些序列以最小更改和高确定性防止失败，即使在随机环境中也是如此。此外，ACTER生成一组多个多样的逆事实序列，使用户能够以最适合其偏好的方式纠正失败。我们还引入了三个多样性指标，可用于评估逆事实序列的多样性。我们在两个RL环境中评估ACTER，这些环境具有离散和连续动作，并表明它可以生成可操作且多样的逆事实序列。我们进行了一项用户研究，以探索ACTER生成的解释如何帮助用户识别和纠正失败。
英文摘要：
Understanding how failure occurs and how it can be prevented in reinforcement learning (RL) is necessary to enable debugging, maintain user trust, and develop personalized policies. Counterfactual reasoning has often been used to assign blame and understand failure by searching for the closest possible world in which the failure is avoided. However, current counterfactual state explanations in RL can only explain an outcome using just the current state features and offer no actionable recourse on how a negative outcome could have been prevented. In this work, we propose ACTER (Actionable Counterfactual Sequences for Explaining Reinforcement Learning Outcomes), an algorithm for generating counterfactual sequences that provides actionable advice on how failure can be avoided. ACTER investigates actions leading to a failure and uses the evolutionary algorithm NSGA-II to generate counterfactual sequences of actions that prevent it with minimal changes and high certainty even in stochastic environments. Additionally, ACTER generates a set of multiple diverse counterfactual sequences that enable users to correct failure in the way that best fits their preferences. We also introduce three diversity metrics that can be used for evaluating the diversity of counterfactual sequences. We evaluate ACTER in two RL environments, with both discrete and continuous actions, and show that it can generate actionable and diverse counterfactual sequences. We conduct a user study to explore how explanations generated by ACTER help users identify and correct failure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
