XUAT-Copilot:基于大语言模型的多智能体协作系统用于自动化用户验收测试
XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance Testing with Large Language Model
AI总结:
本文提出基于大语言模型的多智能体协作系统XUAT-Copilot,通过动作规划、状态检查与参数选择三个智能体协同生成测试脚本,在微信支付UAT中达到接近人工测试的效果并显著提升Pass@1准确率。
摘要:
在过去几年中,我们一直致力于将微信支付(中国最具影响力的移动支付应用之一)的用户验收测试(UAT)流程自动化,并为此开发了一个名为 XUAT 的系统。然而,当前系统中仍存在一个人力密集型阶段,即测试脚本生成。因此,本文重点关注提升当前系统自动化水平的方法,特别是测试脚本生成阶段。随着近期取得的显著成功,大语言模型(LLMs)在实现类人智能方面展现出巨大潜力,并且越来越多的研究领域将 LLMs 用作自主智能体,以获得类人的决策能力。受这些工作的启发,我们提出了一个由 LLM 驱动的多智能体协作系统,名为 XUAT-Copilot,用于自动化 UAT。所提出的系统主要由三个基于 LLM 的智能体组成,分别负责动作规划、状态检查和参数选择,并包含两个附加模块用于状态感知和用例重写。这些智能体与测试设备交互,以协作方式做出类人决策并生成动作命令。在实验研究中,所提出的多智能体系统达到了与人类测试人员接近的有效性,并且与单智能体架构相比,Pass@1 准确率获得了显著提升。更重要的是,该系统已在微信支付移动应用的正式测试环境中上线,在日常开发工作中节省了大量人力。
英文摘要:
In past years, we have been dedicated to automating user acceptance testing (UAT) process of WeChat Pay, one of the most influential mobile payment applications in China. A system titled XUAT has been developed for this purpose. However, there is still a human-labor-intensive stage, i.e, test scripts generation, in the current system. Therefore, in this paper, we concentrate on methods of boosting the automation level of the current system, particularly the stage of test scripts generation. With recent notable successes, large language models (LLMs) demonstrate significant potential in attaining human-like intelligence and there has been a growing research area that employs LLMs as autonomous agents to obtain human-like decision-making capabilities. Inspired by these works, we propose an LLM-powered multi-agent collaborative system, named XUAT-Copilot, for automated UAT. The proposed system mainly consists of three LLM-based agents responsible for action planning, state checking and parameter selecting, respectively, and two additional modules for state sensing and case rewriting. The agents interact with testing device, make human-like decision and generate action command in a collaborative way. The proposed multi-agent system achieves a close effectiveness to human testers in our experimental studies and gains a significant improvement of Pass@1 accuracy compared with single-agent architecture. More importantly, the proposed system has launched in the formal testing environment of WeChat Pay mobile app, which saves a considerable amount of manpower in the daily development work.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。