GUISpector：一种用于GUI原型中自然语言需求自动验证的MLLM智能体框架
GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes
AI总结：
针对现有GUI测试方法难以应对现代界面复杂性、缺乏可执行反馈等问题，提出GUISpector框架，利用MLLM智能体自动验证GUI原型的自然语言需求，提取可执行反馈并提供集成工具，实验验证了其有效性。
摘要：
图形用户界面（GUI）是交互系统的基础，在早期需求获取的原型化过程中发挥着关键作用。确保GUI实现满足自然语言（NL）需求对于稳健的软件工程至关重要，尤其是在大语言模型（LLM）驱动的编程智能体日益融入开发工作流的当下。现有的GUI测试方法，无论是传统方法还是LLM驱动的方法，往往难以应对现代界面的复杂性，且通常缺乏可执行反馈，也无法与自动化开发智能体有效集成。\n本文提出了GUISpector，这是一种创新框架，利用基于多模态大语言模型（MLLM）的智能体实现GUI原型中自然语言需求的自动验证。首先，GUISpector适配MLLM智能体来解读并落地自然语言需求，使其能够自主规划并执行跨GUI应用的验证轨迹。其次，GUISpector从智能体的验证过程中系统提取详细的自然语言反馈，为开发者提供可执行的洞察，可用于迭代优化GUI产物，或在闭环反馈中直接为基于LLM的代码生成提供信息。第三，我们推出了整合上述功能的集成工具，为从业者提供易用界面，用于监督验证运行、检查智能体推理依据以及管理端到端的需求验证流程。\n我们在包含150项需求的综合数据集上评估了GUISpector，这些需求基于跨多种GUI应用的900条验收标准标注，结果表明该框架能有效检测需求满足与违规情况，凸显了其将可执行反馈无缝融入LLM驱动的自动化开发工作流的潜力。GUISpector的演示视频可在以下链接查看：https://youtu.be/JByYF6BNQeE，视频展示了其核心功能。
英文摘要：
GUIs are foundational to interactive systems and play a pivotal role in early requirements elicitation through prototyping. Ensuring that GUI implementations fulfill NL requirements is essential for robust software engineering, especially as LLM-driven programming agents become increasingly integrated into development workflows. Existing GUI testing approaches, whether traditional or LLM-driven, often fall short in handling the complexity of modern interfaces, and typically lack actionable feedback and effective integration with automated development agents. In this paper, we introduce GUISpector, a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes. First, GUISpector adapts a MLLM agent to interpret and operationalize NL requirements, enabling to autonomously plan and execute verification trajectories across GUI applications. Second, GUISpector systematically extracts detailed NL feedback from the agent's verification process, providing developers with actionable insights that can be used to iteratively refine the GUI artifact or directly inform LLM-based code generation in a closed feedback loop. Third, we present an integrated tool that unifies these capabilities, offering practitioners an accessible interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process. We evaluated GUISpector on a comprehensive set of 150 requirements based on 900 acceptance criteria annotations across diverse GUI applications, demonstrating effective detection of requirement satisfaction and violations and highlighting its potential for seamless integration of actionable feedback into automated LLM-driven development workflows. The video presentation of GUISpector is available at: https://youtu.be/JByYF6BNQeE, showcasing its main capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
