将复杂移动应用程序UI手势的视频记录转换为可回放场景
Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios
摘要：
移动应用的屏幕记录易于获得并捕获与软件开发者有关的大量信息（例如，bug或功能请求），使其成为众包应用反馈的流行机制。因此，这些视频正在成为开发人员必须管理的常见工件。鉴于独特的移动开发限制，包括快速的发布周期和快速发展的平台，用于分析所有类型的丰富软件工件的自动化技术为移动开发人员提供了好处。不幸的是，与其他类型的（文本）工件相比，自动分析屏幕记录存在严重的挑战，这是由于它们的图形性质。为了应对这些挑战，本文介绍了V2S+，这是一种将Android应用使用的视频记录转换为可重播场景的自动化方法。V2S+主要基于计算机视觉技术，并采用最新的目标检测和图像分类解决方案来检测和分类视频中捕获的用户手势，并将其转换为可重播的测试场景。鉴于V2S+采用基于计算机视觉的方法，它适用于混合和原生Android应用程序。我们对V2S+进行了广泛的评估，涉及243个视频，描述了4，028个基于GUI的操作，这些操作是从用户使用功能中收集的，并从90多个流行的原生和混合Android应用程序中复制错误。我们的结果表明，V2S+可以准确地重放场景从屏幕记录，并能够再现$\approx$ 90.2%的顺序动作记录在本机应用程序场景上的物理设备，和$\approx$ 83%的顺序动作记录在混合应用程序场景上的仿真器，都具有较低的开销。与三个行业合作伙伴的案例研究从开发人员的角度说明了V2S+的潜在用途。
英文摘要：
Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$ 90.2% of sequential actions recorded in native application scenarios on physical devices, and $\approx$ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
