arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.02936cs.CV
收藏

M-VADER:一个多模态扩散模型

M-VADER: A Model for Diffusion with Multimodal Context

Aleph Alpha GmbH(Aleph Alpha有限公司);TU Darmstadt(达姆施塔特工业大学)

Samuel Weinbach, Marco Bellagente, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Björn Deiseroth, Koen Oostermeijer, Hannah Teufel, Andres Felipe Cruz-Salinas

摘要:

我们介绍M-VADER:一个扩散模型(DM)的图像生成的输出可以指定使用任意组合的图像和文本。我们将展示如何M-VADER使指定的图像的生成使用图像和文本的组合,以及多个图像的组合。以前,已经引入了许多成功的DM图像生成算法,这些算法使得可以使用文本提示指定输出图像。这些模型的成功的启发,并导致的概念,语言已经开发来描述的视觉环境中的元素,人类发现最重要的,我们介绍了一个嵌入模型密切相关的视觉语言模型。具体来说,我们介绍了嵌入模型S-MAGMA:一个130亿参数的多模态解码器结合组件的自回归视觉语言模型MAGMA和偏见微调语义搜索。

英文摘要:

We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the generation of images specified using combinations of image and text, and combinations of multiple images. Previously, a number of successful DM image generation algorithms have been introduced that make it possible to specify the output image using a text prompt. Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model. Specifically, we introduce the embedding model S-MAGMA: a 13 billion parameter multimodal decoder combining components from an autoregressive vision-language model MAGMA and biases finetuned for semantic search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑