Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose $\texttt{Web2Code}$, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage's HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs' abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code are available at https://github.com/MBZUAI-LLM/web2code.
WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Chongqing University(重庆大学);Hubei University of Automotive Technology(湖北汽车工业学院);Rabbitpre AI;University of Notre Dame(圣母大学)
SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents
National University of Defense Technology(国防科技大学);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学)
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
Department of Computer Science(计算机科学系);University of North Carolina at Chapel Hill(北卡罗来纳大学 Chapel Hill 分校);School of Artificial Intelligence and Data Science(人工智能与数据科学学院);University of Science and Technology of China(中国科学技术大学)
Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
The Chinese University of Hong Kong, China(香港中文大学);Singapore Management University, Singapore(新加坡管理学院);Renmin University of China, China(中国人民大学);Tsinghua University, China(清华大学);Southwest University, China(西南大学)
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Allen Institute for AI(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Emory University(埃默里大学);Carnegie Mellon University(卡内基梅隆大学)
LLMs Meet Multimodal Generation and Editing: A Survey
The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Tencent AI Lab(腾讯AI实验室)
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
College of Computer, National University of Defense Technology(国防科技大学计算机学院);School of Computer and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Freedom AI;National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院);Guangdong University of Technology(广东工业大学)
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)