arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.01417cs.SE
收藏

猫头鹰之眼:通过视觉理解发现UI显示问题

Owl Eyes: Spotting UI Display Issues via Visual Understanding

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, Qing Wang

摘要:

图形用户界面(GUI)在软件应用程序和最终用户之间提供了一个可视化的桥梁,通过它,他们可以相互交互。随着科技的进步和审美的发展,GUI的视觉效果越来越受到人们的关注。然而,这样的GUI复杂性对GUI实现提出了很大的挑战。根据我们对众测错误报告的初步研究,由于软件或硬件兼容性的原因,在不同设备上进行GUI渲染时,总会出现文本重叠、屏幕模糊、图像丢失等显示问题。它们会对应用程序的可用性产生负面影响,导致用户体验不佳。为了检测这些问题,我们提出了一种基于深度学习的新方法OwlEye,用于对GUI屏幕截图的视觉信息进行建模。因此,OwlEye可以检测具有显示问题的GUI,并在给定的GUI中定位问题的详细区域,以指导开发人员修复错误。我们手动构建了一个包含4,470个具有UI显示问题的GUI屏幕截图的大规模标记数据集,并开发了一种基于几何学的数据增强方法来提高OwlEye的性能。评估表明,我们的OwlEye在检测UI显示问题时可以达到85%的准确率和84%的召回率,在定位这些问题时可以达到90%的准确率。我们还使用Google Play和F-droid上流行的Android应用程序对OwlEye进行了评估,并成功发现了57个以前未检测到的UI显示问题,其中26个已经得到确认或修复。

英文摘要:

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate OwlEye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑