arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.09055cs.CV
收藏

SemanticDraw:迈向基于图像扩散模型的实时交互式内容创作

SemanticDraw: Towards Real-Time Interactive Content Creation from Image Diffusion Models

ASRI, Department of ECE(ASRI电子工程系);Interdisciplinary Program in Artificial Intelligence(人工智能跨学科项目);SNU-LG AI Research Center(首尔国立大学SNU-LG人工智能研究中心)

Jaerin Lee, Daniel Sungho Jung, Kanggeon Lee, Kyoung Mu Lee

AI总结:

针对扩散模型区域控制速度慢且不兼容加速方法的问题,提出SemanticDraw框架,通过兼容加速技术并引入多提示流批处理流水线,在单张RTX 2080 Ti上实现0.64秒的低延迟交互式图像创作。

摘要:

我们引入了SemanticDraw,这是一种交互式内容创作的新范式,其中能够根据给定的多个手绘区域近乎实时地生成高质量图像,每个区域编码了预设的语义含义。为了最大化内容创作者的生产力并充分实现他们的艺术想象力,这要求他们的工具同时具备快速的交互界面和细粒度的区域控制。尽管最近的扩散模型具有惊人的生成质量,我们发现现有的区域可控性方法非常缓慢(生成 $512 \ imes 512$ 图像需要52秒),并且与LCM等加速方法不兼容,阻碍了它们在交互式内容创作中的巨大潜力。基于这一观察,我们分两步构建了交互式内容创作的解决方案:(1)我们建立了扩散模型的区域控制与加速技术之间的兼容性,在将推理步数减少 $\ imes 10$ 的同时保持了多提示图像生成的高保真度;(2)我们通过新的多提示流批处理流水线提高了生成吞吐量,从而能够在单个 RTX 2080 Ti GPU 上根据多个基于区域的文本提示实现低延迟生成。我们提出的框架可泛化至任何现有的扩散模型和加速调度器,允许在成熟的图像扩散模型之上构建亚秒级(0.64秒)的图像内容创作应用。我们的项目页面是:https://jaerinlee.com/research/semantic-draw

英文摘要:

We introduce SemanticDraw, a new paradigm of interactive content creation where high-quality images are generated in near real-time from given multiple hand-drawn regions, each encoding prescribed semantic meaning. In order to maximize the productivity of content creators and to fully realize their artistic imagination, it requires both quick interactive interfaces and fine-grained regional controls in their tools. Despite astonishing generation quality from recent diffusion models, we find that existing approaches for regional controllability are very slow (52 seconds for $512 \times 512$ image) while not compatible with acceleration methods such as LCM, blocking their huge potential in interactive content creation. From this observation, we build our solution for interactive content creation in two steps: (1) we establish compatibility between region-based controls and acceleration techniques for diffusion models, maintaining high fidelity of multi-prompt image generation with $\times 10$ reduced number of inference steps, (2) we increase the generation throughput with our new multi-prompt stream batch pipeline, enabling low-latency generation from multiple, region-based text prompts on a single RTX 2080 Ti GPU. Our proposed framework is generalizable to any existing diffusion models and acceleration schedulers, allowing sub-second (0.64 seconds) image content creation application upon well-established image diffusion models. Our project page is: https://jaerinlee.com/research/semantic-draw

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑