PeriGuru：一种基于GUI图像理解与大语言模型提示的外围式机器人移动应用操作助手
PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM
AI总结：
针对特殊群体使用智能手机的困难，本文开发了基于GUI图像理解与LLM提示的外围式机器人移动应用操作助手PeriGuru，测试成功率达81.94%，性能远超基线方法，且代码已开源。
摘要：
智能手机极大丰富了人们的日常学习、沟通与娱乐生活，已成为现代生活的必要组成部分。然而，老年人、残障人士等群体在使用智能手机时面临诸多困难，因此亟需移动应用操作助手（又称移动应用智能体）。考虑到隐私、权限及跨平台兼容性问题，本文设计并开发了PeriGuru——一款基于GUI图像理解与大语言模型（LLM）提示的外围式机器人移动应用操作助手。PeriGuru借助一套计算机视觉技术分析GUI截图图像，利用LLM生成动作决策，再由机械臂执行操作。PeriGuru在测试任务集上的成功率达81.94%，比未采用其GUI图像解析与提示设计的方法高出两倍以上。代码已开源至https://github.com/Z2sJ4t/PeriGuru。
英文摘要：
Smartphones have significantly enhanced our daily learning, communication, and entertainment, becoming an essential component of modern life. However, certain populations, including the elderly and individuals with disabilities, encounter challenges in utilizing smartphones, thus necessitating mobile app operation assistants, a.k.a. mobile app agent. With considerations for privacy, permissions, and cross-platform compatibility issues, we endeavor to devise and develop PeriGuru in this work, a peripheral robotic mobile app operation assistant based on GUI image understanding and prompting with Large Language Model (LLM). PeriGuru leverages a suite of computer vision techniques to analyze GUI screenshot images and employs LLM to inform action decisions, which are then executed by robotic arms. PeriGuru achieves a success rate of 81.94% on the test task set, which surpasses by more than double the method without PeriGuru's GUI image interpreting and prompting design. Our code is available on https://github.com/Z2sJ4t/PeriGuru.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
