arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12682cs.SEcs.AI
收藏

检索增强的测试生成:我们走到了哪一步?

Retrieval-Augmented Test Generation: How Far Are We?

York University(约克大学)

Jiho Shin, Nima Shiri Harzevili, Reem Aleithan, Hadi Hemmati, Song Wang

AI总结:

本研究探索RAG在ML/DL API单元测试生成中的应用,评估多种知识来源和LLM,发现RAG平均提升行覆盖率6.5%,并能检测新缺陷,凸显其提升测试覆盖率的潜力。

摘要:

检索增强生成(RAG)已推进了软件工程任务,但在单元测试生成方面仍未得到充分探索。为弥合这一差距,我们研究了基于RAG的单元测试生成对机器学习(ML/DL)API的有效性,并分析了不同知识来源对其效果的影响。我们考察了三种领域特定的RAG来源:(1)API文档(官方指南),(2)GitHub问题(开发者报告的解决方案),以及(3)StackOverflow问答(社区驱动的解决方案)。我们的研究聚焦于五个广泛使用的基于Python的ML/DL库,即TensorFlow、PyTorch、Scikit-learn、Google JAX和XGBoost,针对最常用的API。我们评估了四种最先进的LLM——GPT-3.5-Turbo、GPT-4o、Mistral MoE 8x22B和Llama 3.1 405B——在三种策略下:基础指令提示、基础RAG和API级RAG。在定量方面,我们评估了语法和动态正确性以及行覆盖率。虽然RAG并未提升正确性,但RAG平均将行覆盖率提高了6.5%。我们发现,GitHub问题通过提供来自各种问题的边缘案例,带来了行覆盖率的最佳提升。我们还发现,这些生成的单元测试有助于检测新缺陷。具体而言,检测到28个缺陷,向开发者报告了24个独特缺陷,其中10个被确认,4个被拒绝,10个等待开发者确认。我们的发现凸显了RAG在单元测试生成中通过精准知识来源提升测试覆盖率的潜力。未来工作应聚焦于识别具有独特程序状态的文档的检索技术,以进一步优化基于RAG的单元测试生成。

英文摘要:

Retrieval Augmented Generation (RAG) has advanced software engineering tasks but remains underexplored in unit test generation. To bridge this gap, we investigate the efficacy of RAG-based unit test generation for machine learning (ML/DL) APIs and analyze the impact of different knowledge sources on their effectiveness. We examine three domain-specific sources for RAG: (1) API documentation (official guidelines), (2) GitHub issues (developer-reported resolutions), and (3) StackOverflow Q&As (community-driven solutions). Our study focuses on five widely used Python-based ML/DL libraries, TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, targeting the most-used APIs. We evaluate four state-of-the-art LLMs -- GPT-3.5-Turbo, GPT-4o, Mistral MoE 8x22B, and Llama 3.1 405B -- across three strategies: basic instruction prompting, Basic RAG, and API-level RAG. Quantitatively, we assess syntactical and dynamic correctness and line coverage. While RAG does not enhance correctness, RAG improves line coverage by 6.5% on average. We found that GitHub issues result in the best improvement in line coverage by providing edge cases from various issues. We also found that these generated unit tests can help detect new bugs. Specifically, 28 bugs were detected, 24 unique bugs were reported to developers, ten were confirmed, four were rejected, and ten are awaiting developers' confirmation. Our findings highlight RAG's potential in unit test generation for improving test coverage with well-targeted knowledge sources. Future work should focus on retrieval techniques that identify documents with unique program states to optimize RAG-based unit test generation further.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑