arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.00618cs.CV
收藏

De-Diffusion 使文本成为强大的跨模态接口

De-Diffusion Makes Text a Strong Cross-Modal Interface

Google DeepMind(谷歌DeepMind);Johns Hopkins University(约翰斯·霍普金斯大学)

Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, Jiahui Yu

AI总结:

本文提出 De-Diffusion 方法,将图像编码为文本并通过预训练扩散模型解码重建,使文本成为可解释且灵活的跨模态接口,在多种多模态任务上取得最优性能。

摘要:

我们证明文本是一种强大的跨模态接口。不同于依赖深度嵌入来连接图像和语言作为接口表示的方法,我们的方法将图像表示为文本,从而获得自然语言固有的可解释性和灵活性。我们采用一个自编码器,该自编码器使用预训练的文本到图像扩散模型进行解码。编码器经过训练,将输入图像转换为文本,然后将该文本输入固定的文本到图像扩散解码器以重建原始输入——我们将这一过程称为 De-Diffusion。实验验证了 De-Diffusion 文本表示图像的精确性和全面性,使其能够被现成的文本到图像工具和大语言模型(LLMs)直接用于多种多模态任务。例如,单个 De-Diffusion 模型可以泛化为不同的文本到图像工具提供可迁移的提示,并且通过简单地用少样本示例提示大语言模型,在开放式视觉-语言任务上取得了新的最优结果。

英文摘要:

We demonstrate text as a strong cross-modal interface. Rather than relying on deep embeddings to connect image and language as the interface representation, our approach represents an image as text, from which we enjoy the interpretability and flexibility inherent to natural language. We employ an autoencoder that uses a pre-trained text-to-image diffusion model for decoding. The encoder is trained to transform an input image into text, which is then fed into the fixed text-to-image diffusion decoder to reconstruct the original input -- a process we term De-Diffusion. Experiments validate both the precision and comprehensiveness of De-Diffusion text representing images, such that it can be readily ingested by off-the-shelf text-to-image tools and LLMs for diverse multi-modal tasks. For example, a single De-Diffusion model can generalize to provide transferable prompts for different text-to-image tools, and also achieves a new state of the art on open-ended vision-language tasks by simply prompting large language models with few-shot examples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑