Android in the Zoo: 面向GUI智能体的行动链思维（Chain-of-Action-Thought）
Android in the Zoo: Chain-of-Action-Thought for GUI Agents
AI总结：
针对现有智能手机GUI智能体忽略中间操作语义的问题，提出行动链思维CoAT方法，同时构建含1.8万余对标注的AitZ数据集，实现小模型微调媲美大模型的效果。
摘要：
大语言模型（LLM）推动了智能手机自主GUI智能体的快速发展，这类智能体通过预测一系列API动作，完成自然语言触发的任务。尽管该任务高度依赖过往动作和视觉观测，现有研究通常很少考虑中间截图和屏幕操作所承载的语义信息。为解决这一问题，本研究提出Chain-of-Action-Thought（简称CoAT，行动链思维）方法，该方法会输入先前动作的描述、当前屏幕界面，更关键的是输入关于应执行何种动作以及所选动作将带来何种结果的动作思考。我们证明，在三种现成的大型多模态模型（LMM）上的零样本设置中，与此前提出的上下文建模方法相比，CoAT显著提升了动作预测效果。为进一步推动该方向的研究，我们构建了Android-In-The-Zoo（AitZ）数据集，包含18643个屏幕-动作对，并附带行动链思维标注。实验表明，在我们的AitZ数据集上微调1B参数规模的模型（即AUTO-UI-base），可达到与CogAgent-Chat-18B相当的性能。
英文摘要：
Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typically consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon three off-the-shelf LMMs, CoAT significantly improves the action prediction compared to previous proposed context modeling. To further facilitate the research in this line, we construct a dataset Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 1B model (i.e. AUTO-UI-base) on our AitZ dataset achieves on-par performance with CogAgent-Chat-18B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
