arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1911.01474cs.HCcs.AIcs.CLcs.CV
收藏

VASTA:一个视觉和视觉辅助的智能手机任务自动化系统

VASTA: A Vision and Language-assisted Smartphone Task Automation System

Samsung AI Research Centre(三星人工智能研究中心)

Alborz Rezazadeh Sereshkeh, Gary Leung, Krish Perumal, Caleb Phillips, Minfan Zhang, Afsaneh Fazly, Iqbal Mohomed

摘要:

我们提出了VASTA,一种新颖的视觉和语言辅助编程演示(PBD)系统,用于智能手机任务自动化。一个健壮的PBD自动化系统的开发需要克服三个关键的挑战:第一,如何使一个特定的演示鲁棒的位置和视觉变化的用户界面(UI)元素;第二,如何识别的自动化参数的变化,使演示尽可能普遍化;以及第三,如何从用户的话语识别什么自动化的用户希望执行。为了解决第一个挑战,VASTA利用最先进的计算机视觉技术,包括目标检测和光学字符识别,准确地标记用户演示的交互,而不依赖于底层UI结构。为了解决第二和第三个挑战,VASTA利用高级自然语言理解算法来分析用户话语以触发VASTA自动化脚本,并确定用于泛化的自动化参数。我们运行了一个初始的用户研究,展示了VASTA在聚类用户话语,理解自动化参数的变化,检测所需的UI元素,以及最重要的是,自动化各种任务方面的有效性。该系统的演示视频可在此处获得:http://y2u.be/kr2xE-FixjI

英文摘要:

We present VASTA, a novel vision and language-assisted Programming By Demonstration (PBD) system for smartphone task automation. Development of a robust PBD automation system requires overcoming three key challenges: first, how to make a particular demonstration robust to positional and visual changes in the user interface (UI) elements; secondly, how to recognize changes in the automation parameters to make the demonstration as generalizable as possible; and thirdly, how to recognize from the user utterance what automation the user wishes to carry out. To address the first challenge, VASTA leverages state-of-the-art computer vision techniques, including object detection and optical character recognition, to accurately label interactions demonstrated by a user, without relying on the underlying UI structures. To address the second and third challenges, VASTA takes advantage of advanced natural language understanding algorithms for analyzing the user utterance to trigger the VASTA automation scripts, and to determine the automation parameters for generalization. We run an initial user study that demonstrates the effectiveness of VASTA at clustering user utterances, understanding changes in the automation parameters, detecting desired UI elements, and, most importantly, automating various tasks. A demo video of the system is available here: http://y2u.be/kr2xE-FixjI

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑