arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.05070cs.SEcs.AI
收藏

PromptPex: 语言模型提示的自动测试生成

PromptPex: Automatic Test Generation for Language Model Prompts

University of Washington(华盛顿大学);Microsoft Research(微软研究院)

Reshabh K Sharma, Jonathan De Halleux, Shraddha Barke, Dan Grossman, Benjamin Zorn

AI总结:

PromptPex 是一个基于 LLM 的工具,用于自动为语言模型提示生成和评估单元测试,以提高提示的稳健性和可预测性。

摘要:

大型语言模型(LLMs)被应用于许多领域,这些模型的提示被集成到软件应用中作为类似代码的 artifact。这些提示的行为类似于传统软件,它们接受输入、生成输出并执行特定功能。然而,提示与传统代码有许多不同之处,需要新的方法来确保其稳健性。例如,与传统软件不同,提示的输出取决于解释它的 AI 模型。此外,虽然自然语言提示易于修改,但更新的影响更难预测。因此,需要新的方法来测试、调试和修改运行这些模型的提示。为了解决这些问题,我们开发了 PromptPex,一个基于 LLM 的工具,用于自动为给定提示生成和评估单元测试。PromptPex 从提示中提取输入和输出规范,并利用这些规范生成多样、针对性强且有效的单元测试。这些测试在提示更改时有助于识别回归问题,并且还为理解不同模型如何解释提示提供了工具。我们使用 PromptPex 为八个基准提示生成测试,并通过查看这些测试是否能导致四个不同模型生成无效输出来评估生成测试的质量。PromptPex 生成的测试在导致更多无效模型输出方面优于精心构建的基线 LLM 基于测试生成器。此外,通过从输入提示中提取具体的规范,PromptPex 允许提示编写者清楚地理解和测试其提示的特定方面。PromptPex 的源代码可在 https://github.com/microsoft/promptpex 上获得。

英文摘要:

Large language models (LLMs) are being used in many applications and prompts for these models are integrated into software applications as code-like artifacts. These prompts behave much like traditional software in that they take inputs, generate outputs, and perform some specific function. However, prompts differ from traditional code in many ways and require new approaches to ensure that they are robust. For example, unlike traditional software the output of a prompt depends on the AI model that interprets it. Also, while natural language prompts are easy to modify, the impact of updates is harder to predict. New approaches to testing, debugging, and modifying prompts with respect to the model running them are required. To address some of these issues, we developed PromptPex, an LLM-based tool to automatically generate and evaluate unit tests for a given prompt. PromptPex extracts input and output specifications from a prompt and uses them to generate diverse, targeted, and valid unit tests. These tests are instrumental in identifying regressions when a prompt is changed and also serve as a tool to understand how prompts are interpreted by different models. We use PromptPex to generate tests for eight benchmark prompts and evaluate the quality of the generated tests by seeing if they can cause each of four diverse models to produce invalid output. PromptPex consistently creates tests that result in more invalid model outputs than a carefully constructed baseline LLM-based test generator. Furthermore, by extracting concrete specifications from the input prompt, PromptPex allows prompt writers to clearly understand and test specific aspects of their prompts. The source code of PromptPex is available at https://github.com/microsoft/promptpex.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑