结合TSL与LLM自动化REST API测试：一项比较研究
Combining TSL and LLM to Automate REST API Testing: A Comparative Study
AI总结：
本研究提出RestTSLLM方法，结合TSL与LLM自动化REST API测试生成，评估多个LLM，发现Claude 3.5 Sonnet在成功率、覆盖率和变异得分上最优，验证了LLM自动化测试的潜力。
摘要：
REST API测试的有效执行仍然是开发团队面临的重大挑战，这源于分布式系统的固有复杂性、众多可能的场景以及测试设计可用时间的有限性。对所有输入组合进行穷举测试是不切实际的，往往导致未检测到的故障、高人工投入和有限的测试覆盖率。为解决这些问题，我们引入了RestTSLLM，一种使用测试规范语言（TSL）结合大型语言模型（LLM）来自动化生成REST API测试用例的方法。该方法针对两个核心挑战：测试场景的创建和适当输入数据的定义。所提出的解决方案将提示工程技术与自动化流水线相结合，以评估各种LLM从OpenAPI规范生成测试的能力。评估侧重于成功率、测试覆盖率和变异得分等指标，从而能够系统地比较模型性能。结果表明，表现最佳的LLM——Claude 3.5 Sonnet（Anthropic）、Deepseek R1（Deepseek）、Qwen 2.5 32b（阿里巴巴）和Sabia 3（Maritaca）——始终生成稳健且上下文连贯的REST API测试。其中，Claude 3.5 Sonnet在所有指标上均优于其他所有模型，在本研究中成为最适合此任务的模型。这些发现凸显了LLM基于API规范自动化生成测试的潜力。
英文摘要：
The effective execution of tests for REST APIs remains a considerable challenge for development teams, driven by the inherent complexity of distributed systems, the multitude of possible scenarios, and the limited time available for test design. Exhaustive testing of all input combinations is impractical, often resulting in undetected failures, high manual effort, and limited test coverage. To address these issues, we introduce RestTSLLM, an approach that uses Test Specification Language (TSL) in conjunction with Large Language Models (LLMs) to automate the generation of test cases for REST APIs. The approach targets two core challenges: the creation of test scenarios and the definition of appropriate input data. The proposed solution integrates prompt engineering techniques with an automated pipeline to evaluate various LLMs on their ability to generate tests from OpenAPI specifications. The evaluation focused on metrics such as success rate, test coverage, and mutation score, enabling a systematic comparison of model performance. The results indicate that the best-performing LLMs - Claude 3.5 Sonnet (Anthropic), Deepseek R1 (Deepseek), Qwen 2.5 32b (Alibaba), and Sabia 3 (Maritaca) - consistently produced robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric, emerging in this study as the most suitable model for this task. These findings highlight the potential of LLMs to automate the generation of tests based on API specifications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
