AI 中文总结
研究针对OpenAPI规范常存在的问题，引入故障分类法，在不同严重级别注入故障，用三种测试工具在两个微服务基准上评估，通过多种指标衡量故障影响，揭示规范故障对黑盒测试有效性的作用及相关评估的不足。
AI 中文摘要
OpenAPI规范是微服务系统中黑盒测试工具的主要输入，但先前工作表明这些规范往往不完整、不一致或不正确。多数基于OpenAPI的黑盒测试研究都假定规范正确并评估工具性能。本文引入基于文献的六种OpenAPI规范故障类别的分类法，在五个严重级别注入故障，用EvoMaster、RESTler和Schemathesis三种测试工具在TrainTicket和SocialNetwork两个微服务基准上评估变异规范。通过代码覆盖、规范覆盖、请求/响应质量和行为多样性衡量故障影响。结果表明规范故障在测试工具和系统中导致强烈且异质的降级模式。方法语义中的故障对所有指标造成广泛降级，而其他故障如响应代码修改影响较弱。模式约束的放宽导致隐藏降级，对代码和规范覆盖无影响，但对请求/响应质量影响大。这些发现表明规范质量直接影响黑盒API测试有效性，仅代码和规范覆盖评估可能低估规范故障对微服务系统黑盒测试的影响，应辅以请求/响应质量和行为多样性评估。
英文摘要
OpenAPI specifications are the primary input for black-box testing tools in microservice systems (MSS), yet prior work shows these specifications are often incomplete, inconsistent, or incorrect. Despite this, most studies on OpenAPI-based black-box testing assume correct specifications and evaluate tool performance. We address this gap by introducing a literature-grounded taxonomy of six OpenAPI specification fault classes. We inject faults at five severity levels, and evaluate the resulting mutated specifications on two microservice benchmarks, TrainTicket and SocialNetwork, using three testing tools: EvoMaster, RESTler, and Schemathesis. We measure the impact of these faults using code coverage, specification coverage, request/response quality, and behavioral diversity. Our results show that specification faults cause strong and heterogeneous degradation patterns across testing tools and systems. Faults in method semantics cause broad degradation across all metrics, while others, such as modifications to response codes, remain weak. Relaxations of schema constraints cause hidden degradation, with no impact on code and specification coverage but a large impact on request/response quality. These findings demonstrate that specification quality directly shapes black-box API testing effectiveness. Also, code and specification coverage-only evaluations can understate the impact of specification faults on black-box testing in MSS and should be complemented by request/response quality and behavioral diversity.
CommentsAccepted at the 37th IEEE International Symposium on Software Reliability Engineering (ISSRE 2026)
