布局和图像识别推动跨平台自动化移动测试
Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing
摘要：
碎片化问题已经从Android扩展到不同的平台，例如iOS，移动网络，甚至一些应用程序（app）中的小程序。在这种情况下，记录和回放测试脚本是一种流行的自动化移动应用程序测试方法。但这种方法在跨平台时遇到了严重的问题。需要开发同一应用程序的不同版本，以支持依赖于不同平台支持的不同平台。因此，移动应用程序开发人员需要针对完全相同的测试需求为多个平台开发和维护测试脚本，这大大增加了测试成本。然而，我们发现开发人员在不同平台上为同一应用程序的版本采用高度相似的用户界面布局。这种现象激励我们从类似UI布局的角度重放测试脚本。我们提出了一个图像驱动的移动应用程序测试框架，利用Widget特征匹配和布局特征匹配。我们利用计算机视觉技术对应用截图进行UI特征对比和布局层次提取，获取具有丰富上下文信息的UI结构，包括坐标、相对关系等。基于获取的UI结构，我们可以形成平台无关的测试脚本，然后定位待测目标widget。因此，建议的框架非侵入性重放测试脚本根据一种新的平台无关的测试脚本模型。我们还设计并实现了一个名为LIT的工具，将所提出的框架付诸实践，在此基础上，我们进行了实证研究，以评估所提出的测试框架的有效性和可用性。结果显示，在Android上，整体重放准确率达到63.39%左右（比最先进的方法提高了14%），在iOS上达到21.83%（比最先进的方法提高了98%）。
英文摘要：
The fragmentation problem has extended from Android to different platforms, such as iOS, mobile web, and even mini-programs within some applications (app). In such a situation, recording and replaying test scripts is a popular automated mobile app testing approaches. But such approach encounters severe problems when crossing platforms. Different versions of the same app need to be developed to support different platforms relying on different platform supports. Therefore, mobile app developers need to develop and maintain test scripts for multiple platforms aimed at completely the same test requirements, greatly increasing testing costs. However, we discover that developers adopt highly similar user interface layouts for versions of the same app on different platforms. Such a phenomenon inspires us to replay test scripts from the perspective of similar UI layouts. We propose an image-driven mobile app testing framework, utilizing Widget Feature Matching and Layout Characterization Matching. We use computer vision technologies to perform UI feature comparison and layout hierarchy extraction on app screenshots to obtain UI structures with rich contextual information, including coordinates, relative relationship, etc. Based on acquired UI structures, we can form a platform-independent test script, and then locate the target widgets under test. Thus, the proposed framework non-intrusively replays test scripts according to a novel platform-independent test script model. We also design and implement a tool named LIT to devote the proposed framework into practice, based on which, we conduct an empirical study to evaluate the effectiveness and usability of the proposed testing framework. Results show that the overall replay accuracy reaches around 63.39% on Android (14% improvement over state-of-the-art approaches) and 21.83% on iOS (98% improvement over state-of-the-art approaches).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
