arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.20098cs.CVcs.AIcs.CL
收藏

Web2Code:面向多模态LLM的大规模网页到代码数据集与评估框架

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

MBZUAI(穆罕默德·本·扎耶德人工智能大学);HYU ERICA(汉阳大学ERICA校区);Pengcheng Laboratory(鹏城实验室);CMU(卡内基梅隆大学);UIUC(伊利诺伊大学厄巴纳-香槟分校);Petuum(Petuum公司);SYSU(中山大学)
Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav N… 展开作者

Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, Zhiqiang Shen

AI总结:

针对多模态大语言模型在网页理解和HTML代码生成上的不足,提出Web2Code基准,包含大规模网页到代码数据集与评估框架,显著提升了模型的网页理解及代码生成能力并惠及通用视觉领域。

摘要:

多模态大语言模型(MLLMs)在图像、视频和音频等多种模态的理解和生成任务中取得了显著成功。然而,当前的MLLMs在理解网页截图并生成相应HTML代码方面表现极差。为解决此问题,我们提出Web2Code,这是一个基准,包含用于指令微调的新大规模网页到代码数据集,以及评估MLLMs网页理解和HTML代码转换能力的框架。在数据集构建方面,我们利用预训练LLMs增强现有网页到代码数据集,并生成多样化的新网页池渲染成图像。具体而言,输入为网页图像和指令,响应为网页的HTML代码。我们进一步在响应中包含关于网页内容的多样化自然语言问答对,以实现对网络内容的更全面理解。为评估模型在这些任务中的表现,我们开发了评估框架来测试MLLMs在网页理解和网页到代码生成方面的能力。大量实验表明,我们提出的数据集不仅有益于我们提出的任务,在通用视觉领域也有益处。我们希望我们的工作将有助于开发适用于基于网络内容生成和任务自动化的通用MLLMs。我们的数据和代码可在https://github.com/MBZUAI-LLM/web2code获取。

英文摘要:

Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose $\texttt{Web2Code}$, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage's HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs' abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code are available at https://github.com/MBZUAI-LLM/web2code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑