图形用户界面的目标检测:老式或深度学习或组合?
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?
摘要:
检测GUI图像中的图形用户界面(GUI)元素是特定于域的目标检测任务。它支持许多软件工程任务,例如GUI动画和测试、GUI搜索和代码生成。现有的GUI元素检测研究直接借用计算机视觉(CV)领域的成熟方法,包括依赖于传统图像处理特征(例如,canny edge、contours)和深度学习模型,这些模型学习从大规模GUI数据中进行检测。不幸的是,这些CV方法最初并没有意识到GUI和GUI元素的独特特性以及GUI元素检测任务的高定位精度。我们进行了第一次大规模的实证研究,七个代表性的GUI元素检测方法超过50k GUI图像,以了解这些方法的能力,局限性和有效的设计。这项研究不仅揭示了要解决的技术挑战,但也通知新的GUI元素检测方法的设计。我们针对GUI设计了一种新的非文本GUI元素检测方法,该方法采用自上而下的粗到细策略,并将其与成熟的深度学习模型相结合用于GUI文本检测。我们对25,000张GUI图像的评估表明,我们的方法显著提高了GUI元素检测的最新性能。
英文摘要:
Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。