解锁通过WebSight数据集将网页截图转换为HTML代码
Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset
AI总结:
本文提出WebSight数据集,含200万对HTML代码与截图,微调视觉语言模型实现截图转HTML,并开源以推动研究。
摘要:
在Web开发中使用视觉语言模型(VLMs)提供了一种有前景的策略,以提高效率并解锁无代码解决方案:通过提供UI的截图或草图,VLM可以生成代码来重现它,例如使用HTML等语言。尽管VLM在各种任务上取得了进展,但将截图转换为相应HTML这一具体挑战却鲜有探索。我们认为这主要是由于缺乏合适的高质量数据集。本工作引入了WebSight,一个由200万对HTML代码及其对应截图组成的合成数据集。我们在我们的数据集上微调了一个基础VLM,并展示了其在将网页截图转换为功能性HTML代码方面的熟练程度。为了加速该领域的研究,我们开源了WebSight。
英文摘要:
Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。