Emu：多模态中的生成式预训练
Emu: Generative Pretraining in Multimodality
AI总结：
本文提出基于 Transformer 的多模态基础模型 Emu，通过统一自回归目标端到端建模交错图像、文本与视频，在零样本/少样本多模态任务及指令微调助手上取得优异表现。
摘要：
我们提出 Emu，这是一个基于 Transformer 的多模态基础模型，能够在多模态语境中无缝生成图像和文本。这一杂食性模型通过“一个模型处理全部”的自回归训练过程，可以不加区分地接收任何单模态或多模态数据输入（例如交错排列的图像、文本和视频）。首先，视觉信号被编码为嵌入，并与文本标记共同构成交错输入序列。随后，Emu 以统一目标进行端到端训练，即在多模态序列中分类下一个文本标记，或回归下一个视觉嵌入。这种通用多模态能力使模型能够大规模探索多样化的预训练数据源，例如帧与文本交错的视频、图像与文本交错的网页，以及网络规模的图像-文本对和视频-文本对。Emu 可作为面向图生文和文生图任务的通用多模态接口，并支持上下文中的图像和文本生成。在包括图像描述、视觉问答、视频问答和文生图在内的广泛零样本/少样本任务中，Emu 相比最先进的大型多模态模型展现出卓越性能。论文还展示了通过指令微调获得的多模态助手等扩展能力，其表现同样令人印象深刻。
英文摘要：
We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, and supports in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
