arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.20737cs.CVcs.AIcs.CL
收藏

CANVAS:基于工具的用户界面设计的视觉-语言模型基准

CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design

Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, Juho Kim

AI总结:

CANVAS是一个用于评估视觉-语言模型在基于工具的用户界面设计任务中性能的基准,通过复制和修改UI设计任务来测试模型的工具调用能力和设计质量。

摘要:

用户界面(UI)设计是一个迭代过程,设计师通过诸如Figma或Sketch等设计软件逐步完善其作品。最近,视觉-语言模型(VLMs)在工具调用方面的进展表明,这些模型能够通过迭代操作设计软件来编辑UI设计。理解并增强这种能力很重要,因为它突显了VLMs在传统软件中与设计师协作的潜力。然而,由于没有现有的基准评估基于工具的设计性能,这种能力仍然未知。为了解决这个问题,我们引入了CANVAS,这是一个用于基于工具的用户界面设计的VLMs基准。我们的基准包含598个基于工具的设计任务,这些任务与从3.3千个移动UI设计中采样的30个功能类别(例如,引导、消息)中的地面真实参考配对。在每个任务中,VLM通过基于上下文的工具调用(例如,创建一个矩形作为按钮背景)逐步更新设计,并与设计软件链接。具体而言,CANVAS包含两种任务类型:(i)设计复制评估复制整个UI屏幕的能力;(ii)设计修改评估修改现有屏幕特定部分的能力。结果表明,领先模型表现出更策略性的工具调用,提高了设计质量。此外,我们识别了模型表现出的常见错误模式,指导了未来在增强基于工具的设计能力方面的研究。

英文摘要:

User interface (UI) design is an iterative process in which designers progressively refine their work with design software such as Figma or Sketch. Recent advances in vision language models (VLMs) with tool invocation suggest these models can operate design software to edit a UI design through iteration. Understanding and enhancing this capacity is important, as it highlights VLMs' potential to collaborate with designers within conventional software. However, as no existing benchmark evaluates tool-based design performance, the capacity remains unknown. To address this, we introduce CANVAS, a benchmark for VLMs on tool-based user interface design. Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs across 30 function-based categories (e.g., onboarding, messaging). In each task, a VLM updates the design step-by-step through context-based tool invocations (e.g., create a rectangle as a button background), linked to design software. Specifically, CANVAS incorporates two task types: (i) design replication evaluates the ability to reproduce a whole UI screen; (ii) design modification evaluates the ability to modify a specific part of an existing screen. Results suggest that leading models exhibit more strategic tool invocations, improving design quality. Furthermore, we identify common error patterns models exhibit, guiding future work in enhancing tool-based design capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑