arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.01508cs.SEcs.AI
收藏

COMET:面向深度学习库测试的覆盖引导模型生成

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

The Hong Kong University of Science and Technology(香港科技大学);Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);University of Waterloo(滑铁卢大学);Huazhong University of Science and Technology(华中科技大学)

Meiziniu Li, Jialun Cao, Yongqiang Tian, Tsz On Li, Ming Wen, Shing-Chi Cheung

AI总结:

针对现有 DL 模型生成技术层 API 覆盖不足的问题,本文提出 COMET,利用变异算子、覆盖引导搜索与模型综合提升测试多样性,并在八个 DL 库中发现 32 个新缺陷。

摘要:

近期的深度学习(DL)应用大多构建在 DL 库之上。这些库的质量保证对于 DL 应用的可靠部署至关重要。已有研究提出了多种技术来生成各类 DL 模型,并使用这些模型测试 DL 库。然而,其测试有效性受到所生成 DL 模型中层 API 调用多样性的限制。我们的研究表明,这些技术最多只能覆盖 34.1% 的层输入、25.9% 的层参数值以及 15.6% 的层序列。因此,我们发现,许多由特定层 API 调用(即特定层输入、参数值或层序列)引发的缺陷会被现有技术遗漏。由于这一局限,我们提出 COMET,用于有效生成具有多样化层 API 调用的 DL 模型,以进行 DL 库测试。COMET:(1)设计了一组变异算子和一种基于覆盖的搜索算法,以丰富 DL 模型中层输入、层参数值和层序列的多样性;(2)提出了一种模型综合方法,在不损害层 API 调用多样性的情况下提升测试效率。我们的评估结果显示,COMET 优于基线方法:与当前最先进技术相比,其覆盖的层输入(69.7% 对 34.1%)、层参数值(50.2% 对 25.9%)和层序列(39.0% 对 15.6%)均达到后者的两倍。此外,COMET 比现有技术多覆盖 3.4% 的库分支。最后,COMET 在八个流行 DL 库(包括 TensorFlow 和 MXNet)的最新版本中检测到 32 个新缺陷,其中 21 个已获 DL 库开发者确认,并且这些已确认缺陷中有 7 个已由开发者修复。

英文摘要:

Recent deep learning (DL) applications are mostly built on top of DL libraries. The quality assurance of these libraries is critical to the dependable deployment of DL applications. Techniques have been proposed to generate various DL models and apply them to test these libraries. However, their test effectiveness is constrained by the diversity of layer API calls in their generated DL models. Our study reveals that these techniques can cover at most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer sequences. As a result, we find that many bugs arising from specific layer API calls (i.e., specific layer inputs, parameter values, or layer sequences) can be missed by existing techniques. Because of this limitation, we propose COMET to effectively generate DL models with diverse layer API calls for DL library testing. COMET: (1) designs a set of mutation operators and a coverage-based search algorithm to diversify layer inputs, layer parameter values, and layer sequences in DL models. (2) proposes a model synthesis method to boost the test efficiency without compromising the layer API call diversity. Our evaluation result shows that COMET outperforms baselines by covering twice as many layer inputs (69.7% vs. 34.1%), layer parameter values (50.2% vs. 25.9%), and layer sequences (39.0% vs. 15.6%) as those by the state-of-the-art. Moreover, COMET covers 3.4% more library branches than those by existing techniques. Finally, COMET detects 32 new bugs in the latest version of eight popular DL libraries, including TensorFlow and MXNet, with 21 of them confirmed by DL library developers and 7 of those confirmed bugs have been fixed by developers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑