JetFormer：一种面向原始图像与文本的自回归生成模型
JetFormer: An Autoregressive Generative Model of Raw Images and Text
AI总结：
本研究提出无需独立预训练组件的自回归多模态模型JetFormer，通过联合训练归一化流与Transformer实现图像文本的理解与生成，在图像生成质量与理解能力上表现优异，是首个兼具高保真图像生成与强对数似然界的模型。
摘要：
移除建模约束并统一跨领域架构是近期大型多模态模型训练进展的核心驱动力。然而，这类模型大多仍依赖诸多独立训练的组件，例如模态专属的编码器与解码器。本研究进一步简化了图像与文本的联合生成建模流程，提出一种仅含解码器的自回归Transformer——JetFormer，该模型经训练可直接最大化原始数据的似然，无需依赖任何独立预训练的组件，且能够理解并生成文本与图像两种模态内容。具体而言，我们利用normalizing flow（归一化流）模型得到软令牌图像表示，该表示与自回归多模态Transformer联合训练。在推理阶段，归一化流模型既充当感知任务的图像编码器，也承担图像生成任务的图像解码器角色。JetFormer的文本生成图像质量可与近期基于VQ-VAE和VAE的基准模型相媲美，而这些基准模型依赖预训练的图像自编码器，这类自编码器需通过包含感知损失在内的复杂损失组合进行训练。同时，JetFormer展现出稳健的图像理解能力。据我们所知，JetFormer是首个既能生成高保真图像、又能产出优异对数似然界的模型。
英文摘要：
Removing modeling constraints and unifying architectures across domains has been a key driver of the recent progress in training large multimodal models. However, most of these models still rely on many separately trained components such as modality-specific encoders and decoders. In this work, we further streamline joint generative modeling of images and text. We propose an autoregressive decoder-only transformer - JetFormer - which is trained to directly maximize the likelihood of raw data, without relying on any separately pretrained components, and can understand and generate both text and images. Specifically, we leverage a normalizing flow model to obtain a soft-token image representation that is jointly trained with an autoregressive multimodal transformer. The normalizing flow model serves as both an image encoder for perception tasks and an image decoder for image generation tasks during inference. JetFormer achieves text-to-image generation quality competitive with recent VQ-VAE- and VAE-based baselines. These baselines rely on pretrained image autoencoders, which are trained with a complex mixture of losses, including perceptual ones. At the same time, JetFormer demonstrates robust image understanding capabilities. To the best of our knowledge, JetFormer is the first model that is capable of generating high-fidelity images and producing strong log-likelihood bounds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
