arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.11667cs.SE
收藏

利用自动化声明式 UI 代码生成连接设计与开发

Bridging Design and Development with Automated Declarative UI Code Generation

Ting Zhou, Yanjie Zhao, Xinyi Hou, Xiaoyu Sun, Kai Chen, Haoyu Wang

AI总结:

本文提出 DeclarUI,融合 CV、MLLM 与编译器驱动迭代优化,从 UI 设计自动生成声明式代码,在 React Native、Flutter 和 ArkUI 上显著提升覆盖率、视觉相似度与编译成功率。

摘要:

声明式 UI 框架已在移动应用开发中获得广泛采用,带来了代码可读性提升和维护更简便等优势。尽管有这些优势,将 UI 设计转化为可运行代码的过程仍然具有挑战性且耗时。多模态大语言模型(MLLM)的最新进展展现出从用户界面(UI)设计直接生成移动应用代码的前景。然而,将 MLLM 直接应用于该任务受到两方面限制:难以准确识别 UI 组件,以及难以全面捕获交互逻辑。为应对这些挑战,我们提出 DeclarUI,这是一种协同计算机视觉(CV)、MLLM 和由编译器驱动的迭代优化的自动化方法,用于从设计生成并完善声明式 UI 代码。DeclarUI 通过精确的组件分割、用于建模复杂页面间关系的页面转换图(PTG)以及迭代优化,增强视觉保真度、功能完整性和代码质量。在评估中,DeclarUI 在被广泛采用的声明式 UI 框架 React Native 上优于基线方法,实现了 96.8% 的 PTG 覆盖率和 98% 的编译成功率。值得注意的是,DeclarUI 相比最先进 MLLM 展现出显著提升:PTG 覆盖率提高 123%,视觉相似度得分最高提升 55%,编译成功率提高 29%。我们还通过在 Flutter 和 ArkUI 框架上的成功应用,进一步证明了 DeclarUI 的泛化能力。

英文摘要:

Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic. To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI's generalizability through successful applications to Flutter and ArkUI frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑