arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13480cs.SEcs.AI
收藏

基于多样性的LLM应用自适应测试方法

Adaptive Testing for LLM-Based Applications: A Diversity-based Approach

School of Computing KAIST Daejeon, Republic of Korea;Dept. of Computer Science \& Engineering Chalmers University Gothenburg, Sweden

Juyeon Yoon, Robert Feldt, Shin Yoo

AI总结:

本文提出一种基于多样性的自适应测试方法,将自适应随机测试与字符串距离度量结合,用于LLM提示模板测试,在降低测试预算的同时有效发现故障并增加输出多样性。

摘要:

近年来,由大型语言模型(LLM)驱动的软件系统构建热潮催生了多种测试框架的发展,这些框架主要将提示模板视为测试单元。尽管测试输入执行和输出评估带来了显著成本,但这些工具中优化测试套件的策划仍未得到重视,这需要定制化的测试选择或优先级排序策略。在本文中,我们展示了基于多样性的测试技术,如结合适当字符串距离度量的自适应随机测试(ART),可以有效地应用于提示模板的测试。我们提出的自适应测试方法通过基于现有测试套件及其标注结果得出的分数来选择新的测试输入,从而调整了传统ART流程以适应此场景。我们的结果,通过探索多种基于字符串的距离的各种实现获得,证实了我们的方法能够在减少测试预算的情况下发现故障,并促进更多样化输出的生成。

英文摘要:

The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt templates as the unit of testing. Despite the significant costs associated with test input execution and output assessment, the curation of optimized test suites is yet overlooked in these tools, which calls for tailored test selection or prioritization strategies. In this paper, we show that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates. Our proposed adaptive testing approach adjusts the conventional ART process to this context by selecting new test inputs based on scores derived from existing test suite and their labelling results. Our results, obtained using various implementations that explore several string-based distances, confirm that our approach enables the discovery of failures with reduced testing budgets and promotes the generation of more varied outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑