Painter：教自回归语言模型绘制草图
Painter: Teaching Auto-regressive Language Models to Draw Sketches
AI总结：
本文提出 Painter，通过微调预训练 LLM 使其自回归生成画笔笔触，实现文本到草图生成、对象移除及草图对象检测分类。
摘要：
大型语言模型（LLM）在自然语言理解方面取得了巨大进展，并已成功应用于计算机视觉、机器人、强化学习等其他领域。在本研究中，我们将 LLM 应用于图像生成任务，方法是直接生成虚拟画笔笔触来绘制图像。我们提出 Painter，这是一种 LLM，能够以自回归方式生成相应的画笔笔触，从而将文本描述形式的用户提示转换为草图。我们基于在大型文本语料库上预训练的现成 LLM 构建 Painter，在保留语言理解能力的同时，针对新任务对其进行微调。我们创建了一个包含多样化多对象草图并配有文本提示的数据集，覆盖多种对象类型和任务。Painter 能够根据文本描述生成草图、从画布上移除对象，以及检测和分类草图中的对象。尽管这是使用 LLM 进行自回归图像生成的一项前所未有的开创性工作，但其结果非常令人鼓舞。
英文摘要：
Large language models (LLMs) have made tremendous progress in natural language understanding and they have also been successfully adopted in other domains such as computer vision, robotics, reinforcement learning, etc. In this work, we apply LLMs to image generation tasks by directly generating the virtual brush strokes to paint an image. We present Painter, an LLM that can convert user prompts in text description format to sketches by generating the corresponding brush strokes in an auto-regressive way. We construct Painter based on off-the-shelf LLM that is pre-trained on a large text corpus, by fine-tuning it on the new task while preserving language understanding capabilities. We create a dataset of diverse multi-object sketches paired with textual prompts that covers several object types and tasks. Painter can generate sketches from text descriptions, remove objects from canvas, and detect and classify objects in sketches. Although this is an unprecedented pioneering work in using LLMs for auto-regressive image generation, the results are very encouraging.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
