arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.12469cs.CR
收藏

它们效果如何?探索基于大型语言模型的模糊测试驱动生成

How Effective Are They? Exploring Large Language Model Based Fuzz Driver Generation

Cen Zhang, Yaowen Zheng, Mingqiang Bai, Yeting Li, Wei Ma, Xiaofei Xie, Yuekang Li, Limin Sun, Yang Liu

AI总结:

该研究首次深入评估了基于LLM的模糊测试驱动生成,通过对30个C项目、5个模型和6种提示策略的大规模实验,揭示了其在实际应用中的障碍与提升空间,并改进了OSS-Fuzz-Gen项目。

摘要:

基于LLM(大型语言模型)的模糊测试驱动生成是一个极具前景的研究领域。与传统的基于程序分析的方法不同,这种基于文本的方法更具通用性,能够利用各种API使用信息,从而生成对人类读者友好的代码。然而,对于这一方向的基本问题,如其有效性和潜在挑战,目前仍缺乏深入理解。为了弥补这一空白,我们开展了首次深入研究,旨在探讨使用LLM生成有效模糊测试驱动的重要问题。我们的研究包含一个精心整理的数据集,其中包含来自30个广泛使用的C项目的86个模糊测试驱动生成问题。我们设计了六种提示策略,并在五个最先进的LLM上以五种不同的温度设置进行了测试。总体而言,我们的研究评估了736,430个生成的模糊测试驱动,耗费了8.5亿个token(计费超过8,000美元)。此外,我们将LLM生成的驱动程序与工业界使用的驱动程序进行了对比,并进行了广泛的模糊测试实验(3.75 CPU年)。我们的研究发现:- 尽管基于LLM的模糊测试驱动生成是一个有前景的方向,但在实际应用中仍面临若干障碍;- LLM在为具有复杂细节的API生成有效模糊测试驱动时存在困难。提示策略的三个特色设计选择可以带来益处:发出重复查询、带示例查询以及采用迭代查询过程;- 虽然LLM生成的驱动程序能够产生与工业界所用驱动程序相当的模糊测试结果,但仍存在巨大的提升空间,例如扩展所包含的API使用,或集成语义预言机以促进逻辑错误检测。我们的见解已被实施于改进OSS-Fuzz-Gen项目,从而促进工业界中实用的模糊测试驱动生成。

英文摘要:

LLM-based (Large Language Model) fuzz driver generation is a promising research area. Unlike traditional program analysis-based method, this text-based approach is more general and capable of harnessing a variety of API usage information, resulting in code that is friendly for human readers. However, there is still a lack of understanding regarding the fundamental issues on this direction, such as its effectiveness and potential challenges. To bridge this gap, we conducted the first in-depth study targeting the important issues of using LLMs to generate effective fuzz drivers. Our study features a curated dataset with 86 fuzz driver generation questions from 30 widely-used C projects. Six prompting strategies are designed and tested across five state-of-the-art LLMs with five different temperature settings. In total, our study evaluated 736,430 generated fuzz drivers, with 0.85 billion token costs ($8,000+ charged tokens). Additionally, we compared the LLM-generated drivers against those utilized in industry, conducting extensive fuzzing experiments (3.75 CPU-year). Our study uncovered that: - While LLM-based fuzz driver generation is a promising direction, it still encounters several obstacles towards practical applications; - LLMs face difficulties in generating effective fuzz drivers for APIs with intricate specifics. Three featured design choices of prompt strategies can be beneficial: issuing repeat queries, querying with examples, and employing an iterative querying process; - While LLM-generated drivers can yield fuzzing outcomes that are on par with those used in the industry, there are substantial opportunities for enhancement, such as extending contained API usage, or integrating semantic oracles to facilitate logical bug detection. Our insights have been implemented to improve the OSS-Fuzz-Gen project, facilitating practical fuzz driver generation in industry.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑