arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.15917cs.SE
收藏

面向移动测试的任务描述生成测试:多模态推理方法

Towards Test Generation from Task Description for Mobile Testing with Multi-modal Reasoning

Hieu Huynh, Hai Phung, Hao Pham, Tien N. Nguyen, Vu Nguyen

AI总结:

针对Android GUI测试中任务完成判断不准的问题,提出基于多模态推理的多智能体框架VisiDroid,利用屏幕视觉图像辅助判断任务完成状态,准确率达87.3%,相对提升23.5%。

摘要:

在Android GUI测试中,为可重放为测试脚本的任务生成动作序列是常见的做法。从自然语言描述的任务目标中生成动作序列及相应的测试脚本,可以消除手动编写测试脚本的需求。然而,现有基于大型语言模型(LLM)的方法往往难以识别最终动作,要么过早结束,要么在越过最终屏幕后继续执行。在本文中,我们介绍了VisiDroid,一个基于多模态、LLM的多智能体框架,它迭代地确定下一个动作,并利用屏幕的视觉图像来检测任务的完成状态。多模态方法在两个方面显著增强了我们的模型。首先,这种方法使其能够在仅凭文本内容提供误导性的任务完成指示时,避免过早终止任务。此外,视觉输入帮助工具避免在GUI变化不直接影响任务完成功能时(如字体大小或颜色的调整)出现错误。其次,多模态方法还确保工具不会越过最终屏幕,因为最终屏幕可能缺乏明确的任务完成文本指示,但可能显示表明任务完成的视觉元素,这在GUI应用中很常见。我们的评估显示,VisiDroid达到了87.3%的准确率,相对优于最佳基线23.5%。我们还证明,我们的多模态框架结合图像和文本,使LLM能够更好地判断任务何时完成。

英文摘要:

In Android GUI testing, generating an action sequence for a task that can be replayed as a test script is common. Generating sequences of actions and respective test scripts from task goals described in natural language can eliminate the need for manually writing test scripts. However, existing approaches based on large language models (LLM) often struggle with identifying the final action, and either end prematurely or continue past the final screen. In this paper, we introduce VisiDroid, a multi-modal, LLM-based, multi-agent framework that iteratively determines the next action and leverages visual images of screens to detect the task's completeness. The multi-modal approach enhances our model in two significant ways. First, this approach enables it to avoid prematurely terminating a task when textual content alone provides misleading indications of task completion. Additionally, visual input helps the tool avoid errors when changes in the GUI do not directly affect functionality toward task completion, such as adjustments to font sizes or colors. Second, the multi-modal approach also ensures the tool not progress beyond the final screen, which might lack explicit textual indicators of task completion but could display a visual element indicating task completion, which is common in GUI apps. Our evaluation shows that VisiDroid achieves an accuracy of 87.3%, outperforming the best baseline relatively by 23.5%. We also demonstrate that our multi-modal framework with images and texts enables the LLM to better determine when a task is completed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑