arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.15065cs.SE
收藏

测试强化学习中的故障多样性

Testing for Fault Diversity in Reinforcement Learning

Quentin Mazouni, Helge Spieker, Arnaud Gotlieb, Mathieu Acher

AI总结:

本文提出使用质量多样性优化解决强化学习策略测试中的故障多样性问题,通过将QD优化器与最先进框架对比,证明其能更有效地发现多样化的故障。

摘要:

强化学习是解决序列决策问题(包括驾驶汽车和着陆航天器等复杂任务)的首要技术。在软件确认与验证实践中,针对功能故障检测的测试是在已学习的决策模型中建立可信度的一种便捷方式。虽然近期的研究致力于最大化检测到的故障数量,但都没有在搜索更多多样性时考虑故障特征化。我们认为,策略测试不应寻找尽可能多的失败(例如,触发类似汽车碰撞的输入),而应旨在揭示模型中信息量尽可能大且尽可能多样的故障。在本文中,我们探索使用质量多样性优化来解决策略测试中的故障多样性问题。质量多样性(QD)优化是一种用于解决困难的组合优化问题的进化算法,旨在寻找高质量且多样的解决方案。我们定义并解决了将QD优化适应于动作策略测试的潜在挑战。此外,我们在搜索效率和故障多样性方面,将经典的QD优化器与专门用于策略测试的最先进框架进行了比较。结果表明,QD优化虽然概念简单且普遍适用,但能在决策模型中有效地发现更多样化的故障,并得出结论:基于QD的策略测试是一种有前景的方法。

英文摘要:

Reinforcement Learning is the premier technique to approach sequential decision problems, including complex tasks such as driving cars and landing spacecraft. Among the software validation and verification practices, testing for functional fault detection is a convenient way to build trustworthiness in the learned decision model. While recent works seek to maximise the number of detected faults, none consider fault characterisation during the search for more diversity. We argue that policy testing should not find as many failures as possible (e.g., inputs that trigger similar car crashes) but rather aim at revealing as informative and diverse faults as possible in the model. In this paper, we explore the use of quality diversity optimisation to solve the problem of fault diversity in policy testing. Quality diversity (QD) optimisation is a type of evolutionary algorithm to solve hard combinatorial optimisation problems where high-quality diverse solutions are sought. We define and address the underlying challenges of adapting QD optimisation to the test of action policies. Furthermore, we compare classical QD optimisers to state-of-the-art frameworks dedicated to policy testing, both in terms of search efficiency and fault diversity. We show that QD optimisation, while being conceptually simple and generally applicable, finds effectively more diverse faults in the decision model, and conclude that QD-based policy testing is a promising approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑