arXiv 2602.025482026-01-30cs.LGcs.AIcs.CVcs.MA

ToolTok: 为高效且通用的GUI代理的工具标记化

ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

Department of Computer Science, Tsinghua University, Beijing, China（清华大学计算机科学系）；Guangming Laboratory, Shenzhen, China（光明实验室）；National University of Singapore, Singapore（新加坡国立大学）；Peking University, Beijing, China（北京大学）；MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU–BIT University（MSU-BIT-SMBU应用数学联合研究中心）

Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li

备注：: 8 pages main paper, 18 pages total, 8 figures, 5 tables, code at https://github.com/ZephinueCode/ToolTok

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

ToolTok通过多步路径寻找范式，利用语义锚定机制和易到难课程，实现高效且通用的GUI代理，以较少数据获得优异性能。

摘要：

现有的依赖坐标基础的一步视觉定位的GUI代理模型在面对输入分辨率和纵横比变化时难以泛化。替代方案引入了无坐标的策略，但因数据稀缺而学习困难。为了解决这些限制，我们提出了ToolTok，一种新的多步路径寻找范式用于GUI代理，其中操作被建模为一系列逐步的工具使用序列。具体来说，我们设计了符合人类交互习惯的工具，并用可学习的标记嵌入表示每个工具。为了在有限监督下高效地进行嵌入学习，ToolTok引入了语义锚定机制，将每个工具与语义相关概念联系起来作为自然归纳偏置。为进一步使预训练的大语言模型逐步获取工具语义，我们构建了一个由三个任务组成的易到难课程：标记定义问答、纯文本引导的工具选择和简化视觉路径寻找。在多个基准上的广泛实验表明，ToolTok在与之相当的模型规模（4B）中实现了优越的性能，并且在大幅更大的模型（235B）中仍具有竞争力。值得注意的是，这些结果是在其他后训练方法所需训练数据的不到1%的情况下获得的。此外，ToolTok在未见过的场景中表现出强大的泛化能力。我们的训练与推理代码在https://github.com/ZephinueCode/ToolTok上开源。

英文摘要：

Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training & inference code is open-source at https://github.com/ZephinueCode/ToolTok.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。