DragOn:基于拖拽的GUI交互基准与数据集
DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions
AI总结:
针对GUI代理在拖拽操作(如拖放、滑动、高亮)上的性能不足,提出DragOn基准和训练数据集,涵盖文本高亮、单元格选择、元素缩放和滑块操作四个领域,包含28.6万张训练截图和350万个训练任务,评估了多个模型并显示数据集能提升下游任务性能。
摘要:
GUI代理——通过图形用户界面控制桌面、网页浏览器和移动设备的视觉模型——有望自动化广泛的数字任务。虽然百万级数据集在点击定位方面取得了显著进展,但拖拽定位(例如拖放、滑动、高亮)的数据规模仍小一个数量级,当前模型在复杂的基于拖拽的交互上表现不足。我们引入了DragOn,一个拖拽定位基准和训练数据集,涵盖四个领域:文本高亮、单元格选择、元素缩放和滑块操作。该数据集包含28.6万张训练截图和350万个训练任务,外加一个2000个样本的保留评估集。我们评估了专有模型(GPT、Claude)和开源模型(Qwen、Kimi、Holo),以及在我们训练数据上微调的Qwen VLM。结果表明,我们的数据集可以提升最先进模型在下游计算机使用任务上的性能。
英文摘要:
GUI agents - vision-based models that control desktops, web browsers, and mobile devices through graphical user interfaces - promise to automate a wide range of digital tasks. While million-scale datasets have enabled substantial progress on click-grounding, drag grounding (e.g. drag-and-drop, swipe, highlight) data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. We introduce DragOn, a drag grounding benchmark and training dataset covering four domains: text highlighting, cell selection, element resizing and slider manipulation. The dataset comprises 286K training screenshots and 3.5M training tasks, plus a 2000-example held-out evaluation suite. We evaluate proprietary (GPT, Claude) and open-weight (Qwen, Kimi, Holo) models, as well as a Qwen VLM fine-tuned on our training data. Results suggest that our dataset could improve performance of state-of-the-art models on downstream computer-use tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。