迈向可靠的AI：确保自动驾驶汽车系统级测试质量的充分性度量
Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles
AI总结：
针对自动驾驶系统测试充分性不足的问题，提出黑盒TISA度量及可视化框架，通过强相关性和低计算成本有效评估测试套件质量。
摘要：
基于AI的系统已在包括自动驾驶汽车（AVs）在内的各个领域广泛普及。然而，由于其复杂性，确保其可靠性和安全性具有挑战性。旨在评估传统软件测试有效性的常规测试充分性度量，往往不足以或不适用于这些系统。专为这些系统设计的白盒度量利用神经元覆盖信息。这些覆盖度量需要访问底层AI模型和训练数据，而这些数据可能并不总是可用的。此外，现有的充分性度量与生成测试套件中检测故障的能力相关性较弱，这构成了我们旨在弥合的差距。\n在本文中，我们引入了一组黑盒测试充分性度量，称为“测试套件实例空间充分性”（TISA）度量，可用于衡量测试套件的有效性。TISA度量提供了一种评估测试套件多样性和覆盖度以及测试期间检测到的错误范围的方法。此外，我们引入了一个框架，允许测试人员在二维空间中可视化测试套件的多样性和覆盖度，便于识别需要改进的区域。\n我们通过检查TISA度量与自动驾驶汽车系统级仿真测试中检测到的错误数量的相关性来评估其有效性。强相关性加上较短的计算时间，表明其在估计自动驾驶汽车测试充分性方面的有效性和效率。
英文摘要：
AI-powered systems have gained widespread popularity in various domains, including Autonomous Vehicles (AVs). However, ensuring their reliability and safety is challenging due to their complex nature. Conventional test adequacy metrics, designed to evaluate the effectiveness of traditional software testing, are often insufficient or impractical for these systems. White-box metrics, which are specifically designed for these systems, leverage neuron coverage information. These coverage metrics necessitate access to the underlying AI model and training data, which may not always be available. Furthermore, the existing adequacy metrics exhibit weak correlations with the ability to detect faults in the generated test suite, creating a gap that we aim to bridge in this study. In this paper, we introduce a set of black-box test adequacy metrics called "Test suite Instance Space Adequacy" (TISA) metrics, which can be used to gauge the effectiveness of a test suite. The TISA metrics offer a way to assess both the diversity and coverage of the test suite and the range of bugs detected during testing. Additionally, we introduce a framework that permits testers to visualise the diversity and coverage of the test suite in a two-dimensional space, facilitating the identification of areas that require improvement. We evaluate the efficacy of the TISA metrics by examining their correlation with the number of bugs detected in system-level simulation testing of AVs. A strong correlation, coupled with the short computation time, indicates their effectiveness and efficiency in estimating the adequacy of testing AVs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
