测试在发现软件缺陷方面的有效性,不仅取决于测试输入的特性及其对软件的测试 thoroughness( thoroughness 此处为测试覆盖程度),还取决于用于判断软件是否按预期运行的预言机(oracle)的质量。因此,评估预言机质量对提升测试流程的整体有效性至关重要。现有指标已被用于该目的,但它们要么无法为指导预言机改进提供全面依据,要么仅针对特定类型的预言机定制,从而限制了其通用性。\n 本文提出了状态字段覆盖率(state field coverage),这是一种用于评估预言机质量的新型指标。该指标衡量的是,在测试执行过程中,预言机可能访问的、由类字段静态定义的对象状态的比例。我们提出该指标的核心直觉是:状态字段覆盖率更高的预言机更有可能检测到被测软件中的故障,因为它们会检查更大比例的对象状态,以判断测试是否通过。\n 我们实现了一种静态计算状态字段覆盖率指标的机制。由于是静态计算,该指标效率很高,并且能够通过识别尚未被检查的状态字段,为改进测试预言机提供直接指导。我们通过涉及273个表示不变量(representation invariants)和249,027条测试断言的实验对状态字段覆盖率进行了评估。结果表明,状态字段覆盖率是一种非常适合评估预言机质量的指标,因为它与以变异得分(mutation score)衡量的预言机故障检测能力高度相关。
英文摘要:
The effectiveness of testing in uncovering software defects depends not only on the characteristics of the test inputs and how thoroughly they exercise the software, but also on the quality of the oracles used to determine whether the software behaves as expected. Therefore, assessing the quality of oracles is crucial to improve the overall effectiveness of the testing process. Existing metrics have been used for this purpose, but they either fail to provide a comprehensive basis for guiding oracle improvement, or they are tailored to specific types of oracles, thus limiting their generality.
In this paper, we introduce state field coverage, a novel metric for assessing oracle quality. This metric measures the proportion of an object's state, as statically defined by its class fields, that an oracle may access during test execution. The main intuition of our metric is that oracles with a higher state field coverage are more likely to detect faults in the software under analysis, as they inspect a larger portion of the object states to determine whether tests pass or not.
We implement a mechanism to statically compute the state field coverage metric. Being statically computed, the metric is efficient and provides direct guidance for improving test oracles by identifying state fields that remain unexamined. We evaluate state field coverage through experiments involving 273 representation invariants and 249,027 test assertions. The results show that state field coverage is a well-suited metric for assessing oracle quality, as it strongly correlates with the oracles' fault-detection ability, measured by mutation score.
VISOR: A Vision-Language Model-based Test Oracle for Testing Robots
Simula Research Laboratory(Simula研究实验室);Oslo Metropolitan University(奥斯陆理工大学);Mondragon University(蒙dragon大学);National Institute of Informatics(国立信息研究所)
LLM-Based Test Generation: Information Sources, Generation Strategies, and Quality Evidence
Chengdu Institute of Computer Applications, Chinese Academy of Sciences(中国科学院成都计算机应用研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Multidisciplinary Research for Advanced Materials (IMRAM), Tohoku University(东北大学先进材料多学科研究所);Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology(深圳先进技术研究院人工智能研究所)
Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness
Beihang University(北京航空航天大学);National College for Excellent Engineers(卓越工程师学院);Qingdao Research Institute(青岛研究院);Hangzhou Innovation Institute(杭州创新研究院)
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
Mondragon University(蒙得龙大学);Simula Research Laboratory(Simula研究实验室);University of Ottawa(渥太华大学);Research Ireland Lero Centre for Software(爱尔兰Lero软件研究中心)