arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.04893cs.HC
收藏

屏幕识别:从像素为移动应用程序创建辅助功能元数据

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham

摘要:

移动平台上可用的许多辅助功能要求应用程序(app)提供描述用户界面(UI)组件的完整且准确的元数据。不幸的是,许多应用程序没有提供足够的元数据,以使辅助功能按预期工作。在本文中,我们将探索从像素中推断移动应用程序的可访问性元数据,因为视觉界面通常最能反映应用程序的全部功能。我们训练了一个健壮、快速、内存高效的设备模型,使用我们收集和注释的77,637个屏幕(来自4,068个iPhone应用程序)的数据集来检测UI元素。为了进一步改进UI检测并添加语义信息,我们引入了语义分析(例如,UI分组和排序)和附加模型(例如,识别UI内容、状态、交互性)。我们构建了Screen Recognition来生成辅助功能元数据,以增强iOS的旁白功能。在一项针对9名屏幕阅读器用户的研究中,我们验证了我们的方法可以提高现有移动应用程序的可访问性,甚至可以使用以前无法访问的应用程序。

英文摘要:

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app's full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑