arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.04552cs.CVcs.AIcs.LGcs.MM
收藏

生成带插图的指令

Generating Illustrated Instructions

Meta;Columbia University(哥伦比亚大学)

Sachit Menon, Ishan Misra, Rohit Girdhar

AI总结:

本研究提出带插图指令生成新任务,设计评估指标,提出结合大语言模型与扩散模型的StackedDiffusion方法,性能优于基线与现有多模态大模型,30%案例中用户偏好度超人工内容,可支撑多种个性化新应用。

摘要:

我们提出了生成带插图的指令这一新任务,即根据用户需求定制的可视化指令。我们明确了该任务独有的要求,并通过一套自动与人工评估指标将其形式化,这些指标用于衡量生成内容的有效性、一致性和实用性。我们结合大语言模型(LLMs)与强大的文生图扩散模型的优势,提出了一种名为StackedDiffusion的简单方法,该方法可在给定文本输入的情况下生成此类带插图的指令。所得模型性能显著优于基线方法和最先进的多模态大语言模型;在30%的案例中,用户甚至更偏好它而非人工生成的文章。最值得注意的是,它能实现各类新颖且令人兴奋的应用,远超出网络上静态文章所能提供的范围,例如可响应用户个体情况、包含中间步骤与图片的个性化指令。

英文摘要:

We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions given text as input. The resulting model strongly outperforms baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases, users even prefer it to human-generated articles. Most notably, it enables various new and exciting applications far beyond what static articles on the web can provide, such as personalized instructions complete with intermediate steps and pictures in response to a user's individual situation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑