arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.03075cs.SEcs.CR
收藏

基于变异分析的模糊测试工具系统性评估

Systematic Assessment of Fuzzers using Mutation Analysis

Philipp Görz, Björn Mathis, Keno Hassler, Emre Güler, Thorsten Holz, Andreas Zeller, Rahul Gopinath

AI总结:

针对模糊测试工具效果评估难的问题,本文采用可合并多变异的现代变异分析技术,搭建评估基准,首次实现用变异分析评估模糊测试工具,实验发现现有工具检测变异比例低,为后续研究指明方向。

摘要:

模糊测试是发现程序漏洞的重要方法。尽管近年来该领域取得了长足进展,但衡量和比较模糊测试工具的有效性仍是一个开放的研究问题。在软件测试中,评估测试质量的黄金标准是变异分析,它用于评估测试检测合成缺陷的能力:如果一组测试无法检测到这类变异,那么它大概率也无法检测到真实缺陷。变异分析涵盖了多种覆盖率指标,能够提供大量且多样的故障,这些故障的触发和检测难度可任意设定,因此能避免饱和度和过拟合问题。遗憾的是,传统变异分析的成本对于模糊测试而言过高,因为每个变异都需要独立评估。\n在本文中,我们采用了可合并多个变异的现代变异分析技术,首次实现了用变异分析来评估和比较模糊测试工具。我们搭建了一个模糊测试工具评估基准,并将其应用于多款流行的模糊测试工具和测试对象。通过全面评估,我们展示了如何利用该基准评估模糊测试工具的性能,以及衡量技术改进的效果。所需的CPU时间处于可控范围:在7个测试对象、总计141278个变异的场景下,分析一款模糊测试工具需要4.09 CPU年。我们发现,当前的模糊测试工具仅能检测到很小比例的变异,这应被视为未来研究的挑战——尤其是在以下两方面:(1)检测除通用崩溃之外的故障;(2)触发变异(进而触发故障)。

英文摘要:

Fuzzing is an important method to discover vulnerabilities in programs. Despite considerable progress in this area in the past years, measuring and comparing the effectiveness of fuzzers is still an open research question. In software testing, the gold standard for evaluating test quality is mutation analysis, which evaluates a test's ability to detect synthetic bugs: If a set of tests fails to detect such mutations, it is expected to also fail to detect real bugs. Mutation analysis subsumes various coverage measures and provides a large and diverse set of faults that can be arbitrarily hard to trigger and detect, thus preventing the problems of saturation and overfitting. Unfortunately, the cost of traditional mutation analysis is exorbitant for fuzzing, as mutations need independent evaluation. In this paper, we apply modern mutation analysis techniques that pool multiple mutations and allow us -- for the first time -- to evaluate and compare fuzzers with mutation analysis. We introduce an evaluation bench for fuzzers and apply it to a number of popular fuzzers and subjects. In a comprehensive evaluation, we show how we can use it to assess fuzzer performance and measure the impact of improved techniques. The required CPU time remains manageable: 4.09 CPU years are needed to analyze a fuzzer on seven subjects and a total of 141,278 mutations. We find that today's fuzzers can detect only a small percentage of mutations, which should be seen as a challenge for future research -- notably in improving (1) detecting failures beyond generic crashes (2) triggering mutations (and thus faults).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑