利用大语言模型改进网页元素定位
Improving web element localization by using a large language model
AI总结:
针对传统网页元素定位方法无法理解上下文语义的问题,提出结合大语言模型的VON Similo LLM方法,从候选元素中选最优匹配项,实验显示定位失败率降44%,可提升GUI测试自动化的元素识别效果。
摘要:
基于网页的测试自动化高度依赖对网页元素的精准查找。传统方法通过比对属性来实现定位,但无法理解元素与文本的上下文及含义。GPT-4等大语言模型(Large Language Models, LLMs)的出现为软件工程和网页元素定位提供了新的机遇,这类模型在部分任务中展现出类人的推理能力。本文提出并评估了VON Similo LLM——一种增强型网页元素定位方法。该方法利用大语言模型,从现有VON Similo方法识别出的排名靠前的候选元素中,选出最可能匹配的网页元素,理想目标是进一步接近人类级别的选择准确率。研究使用来自48个真实网页应用的804对网页元素开展了实验。我们统计了正确识别的元素数量及执行时间,对比了VON Similo LLM与基线算法的有效性和效率。此外,针对原始方法未能找到正确网页元素的所有案例,我们记录并分析了大语言模型的决策动机。实验结果显示VON Similo LLM性能有所提升:在804个测试样本中,定位失败的数量从70降至39,降幅达44%。尽管使用GPT-4模型会导致执行速度变慢且产生额外成本,但大语言模型的类人推理能力在提升网页元素定位方面展现出应用前景。大语言模型技术可增强GUI测试自动化中的网页元素识别能力,减少误报,并有潜力降低维护成本。不过,要充分理解大语言模型在GUI测试中的能力、局限性及实际应用场景,仍需开展进一步研究。
英文摘要:
Web-based test automation heavily relies on accurately finding web elements. Traditional methods compare attributes but don't grasp the context and meaning of elements and words. The emergence of Large Language Models (LLMs) like GPT-4, which can show human-like reasoning abilities on some tasks, offers new opportunities for software engineering and web element localization. This paper introduces and evaluates VON Similo LLM, an enhanced web element localization approach. Using an LLM, it selects the most likely web element from the top-ranked ones identified by the existing VON Similo method, ideally aiming to get closer to human-like selection accuracy. An experimental study was conducted using 804 web element pairs from 48 real-world web applications. We measured the number of correctly identified elements as well as the execution times, comparing the effectiveness and efficiency of VON Similo LLM against the baseline algorithm. In addition, motivations from the LLM were recorded and analyzed for all instances where the original approach failed to find the right web element. VON Similo LLM demonstrated improved performance, reducing failed localizations from 70 to 39 (out of 804), a 44 percent reduction. Despite its slower execution time and additional costs of using the GPT-4 model, the LLMs human-like reasoning showed promise in enhancing web element localization. LLM technology can enhance web element identification in GUI test automation, reducing false positives and potentially lowering maintenance costs. However, further research is necessary to fully understand LLMs capabilities, limitations, and practical use in GUI testing.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。