arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.06369cs.CVcs.AIcs.SE
收藏

WebCode2M:一个用于从网页设计生成代码的真实世界数据集

WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs

Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Chongqing University(重庆大学);Hubei University of Automotive Technology(湖北汽车工业学院);Rabbitpre AI;University of Notre Dame(圣母大学)

Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Bohua Chen, Dongping Chen, Siyuan Wu, Xing Zhou, Wenbin Jiang, Hai Jin, Xiangliang Zhang

AI总结:

针对多模态大语言模型在网页代码生成中缺乏高质量真实数据的问题,本文提出包含256万实例的WebCode2M数据集,并引入WebCoder基线与TreeBLEU指标,显著提升模型生成能力。

摘要:

从网页设计自动生成网页代码可以显著减少前端开发人员的工作量,而最近的多模态大语言模型(MLLMs)在这一领域显示出巨大的潜力。然而,我们的调查发现,大多数现有的MLLMs受限于缺乏高质量、大规模、真实世界的数据集,导致在自动化网页代码生成方面表现不足。为了填补这一空白,本文介绍了WebCode2M,这是一个包含256万个实例的新数据集,每个实例包含一个设计图像以及相应的网页代码和布局细节。WebCode2M来源于真实世界的网络资源,为各种应用中的网页代码生成提供了丰富且有价值的数据集。数据集的质量通过一个评分模型来保证,该模型会过滤掉存在美学缺陷或其他不完整元素的实例。为了验证WebCode2M的有效性,我们引入了一个基于视觉变换器(ViT)的基线模型,名为WebCoder,并建立了一个用于公平比较的基准。此外,我们引入了一个新的度量标准TreeBLEU,用于衡量结构层次召回率。基准测试结果表明,我们的数据集显著提高了MLLMs从网页设计生成代码的能力,证实了其有效性和在前端设计工具中未来应用的可用性。最后,我们强调了数据集带来的几个实际挑战,呼吁进一步研究。代码和数据集在我们的项目主页上公开可用:https://webcode2m.github.io。

英文摘要:

Automatically generating webpage code from webpage designs can significantly reduce the workload of front-end developers, and recent Multimodal Large Language Models (MLLMs) have shown promising potential in this area. However, our investigation reveals that most existing MLLMs are constrained by the absence of high-quality, large-scale, real-world datasets, resulting in inadequate performance in automated webpage code generation. To fill this gap, this paper introduces WebCode2M, a new dataset comprising 2.56 million instances, each containing a design image along with the corresponding webpage code and layout details. Sourced from real-world web resources, WebCode2M offers a rich and valuable dataset for webpage code generation across a variety of applications. The dataset quality is ensured by a scoring model that filters out instances with aesthetic deficiencies or other incomplete elements. To validate the effectiveness of WebCode2M, we introduce a baseline model based on the Vision Transformer (ViT), named WebCoder, and establish a benchmark for fair comparison. Additionally, we introduce a new metric, TreeBLEU, to measure the structural hierarchy recall. The benchmarking results demonstrate that our dataset significantly improves the ability of MLLMs to generate code from webpage designs, confirming its effectiveness and usability for future applications in front-end design tools. Finally, we highlight several practical challenges introduced by our dataset, calling for further research. The code and dataset are publicly available at our project homepage: https://webcode2m.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑