arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.16232cs.CLcs.AI
收藏

Sketch2Code:评估用于交互式网页设计原型生成的视觉语言模型

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

Stanford University(斯坦福大学);Georgia Tech(佐治亚理工学院)

Ryan Li, Yanzhe Zhang, Diyi Yang

AI总结:

Sketch2Code基准评估视觉语言模型将粗略草图转换为网页原型的能力,发现现有模型表现有限,且用户更偏好智能体主动提问而非被动接收反馈。

摘要:

草图是UI设计师构思早期想法的一种自然且易于使用的媒介。然而,现有的UI/UX自动化研究通常需要高保真输入,如Figma设计或详细截图,这限制了可访问性并阻碍了高效的设计迭代。为弥合这一差距,我们引入了Sketch2Code,这是一个基准测试,用于评估最先进的视觉语言模型(VLMs)在将粗略草图自动转换为网页原型方面的能力。除了端到端基准测试之外,Sketch2Code还支持模拟真实设计工作流程的交互式智能体评估,其中基于VLM的智能体通过与模拟用户通信来迭代地改进其生成结果,要么被动地接收反馈指令,要么主动地提出澄清问题。我们全面分析了十个商业和开源模型,表明Sketch2Code对现有VLMs具有挑战性;即使是最有能力的模型也难以准确解释草图并提出能带来稳定改进的有效问题。尽管如此,一项由UI/UX专家参与的用户研究表明,与被动接收反馈相比,用户显著偏好主动提问,这凸显了开发更有效的多轮对话智能体范式的必要性。

英文摘要:

Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑