端到端视觉分词器调优
End-to-End Vision Tokenizer Tuning
AI总结：
针对视觉分词器与下游任务优化脱节导致的表示瓶颈，提出 ETT 端到端联合调优方法，在多模态理解与生成任务上较冻结基线提升 2-6%。
摘要：
现有的视觉分词将视觉分词器的优化与下游训练隔离开来，隐含地假设视觉 token 能够很好地泛化到各种任务，例如图像生成和视觉问答。为低层重建而优化的视觉分词器对需要多样化表示和语义的下游任务是无感知的。这种解耦范式带来了严重的错位：视觉分词的损失可能成为目标任务的表示瓶颈。例如，对给定图像中的文字进行分词时出现的错误，会导致识别或生成这些文字时结果不佳。为了解决这个问题，我们提出了 ETT，这是一种端到端视觉分词器调优方法，能够实现视觉分词与目标自回归任务之间的联合优化。以往的自回归模型仅使用来自冻结视觉分词器的离散索引，与之不同，ETT 利用分词器码本的视觉嵌入，并结合重建和描述目标对视觉分词器进行端到端优化。ETT 可以通过极少的架构修改无缝集成到现有训练流程中。我们的 ETT 易于实现和集成，无需调整所采用的大型语言模型的原始码本或架构。大量实验表明，我们提出的端到端视觉分词器调优带来了显著的性能提升，即与冻结分词器基线相比，多模态理解和视觉生成任务提升 2-6%，同时保留了原始重建能力。我们希望这种非常简单且强大的方法能够为图像生成和理解之外的多模态基础模型提供支持。
英文摘要：
Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
