从视觉控件到UI代码：高效的基于工具的生成
From Visual Widgets to UI Code: Efficient Tool-Grounded Generation
浏览论文内容
中文总结 AI 辅助
本研究提出轻量级工具框架WidgetGen，在6个多模态模型和1000个控件上，其视觉重建指标优于直接提示和Widget2Code，且重建的图像-代码对可提升Qwen系列模型性能
中文摘要 AI 辅助
现有的截图转代码系统在灵活性和可控性之间存在权衡。直接多模态生成可能会虚构可见细节，而结构化流水线可通过组件分解、预定义模板和定制中间表示减少此类错误，但这些结构会引入额外的生成编排，并将输出限制在表示覆盖的设计范围内。我们研究选择性工具接地能否改善直接控件到代码生成的保真度-效率权衡。我们推出WidgetGen，这是一个轻量级的基于工具的框架，可提取可观察的文本和颜色证据，执行高级布局和可选图表推理，并直接生成可执行的JavaScript XML（JSX）。该设计减少了对组件生成的依赖，同时避免了固定的UI模式。在6个多模态模型和1000个保留的控件上，WidgetGen在大多数视觉重建指标上优于直接提示和结构化Widget2Code流水线，在面积、可读性和风格方面持续提升。最后，通过重建得到的图像-代码对，在监督微调后，可提升6个Qwen系列开放权重模型的所有报告指标。这些结果确立了WidgetGen作为强大的轻量级基线的地位，并表明选择性证据接地是广泛表示约束的有效替代方案。
英文摘要
Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.
发表机构
- McMaster University（麦克马斯特大学）
- University of Toronto（多伦多大学）
- Concordia University（康考迪亚大学）
机构由 AI 辅助整理，请以论文原文为准。
