借助SEED Tokenizer让LLaMA具备视觉理解与绘制能力
Making LLaMA SEE and Draw with SEED Tokenizer
AI总结：
本研究提出图像分词器SEED，通过两大设计原则实现文本与图像在统一自回归Transformer中的互换处理，基于LLaMA构建的SEED-LLaMA在多模态理解生成任务上表现优异，还具备多轮上下文生成等组合式涌现能力。
摘要：
大语言模型（LLMs）的巨大成功拓展了多模态的发展潜力，推动通用人工智能（AGI）逐步演进。真正的AGI智能体不仅应具备执行预设多任务的能力，还应在开放世界场景中展现出涌现能力。然而，尽管近期多模态大语言模型已取得显著进展，它们仍无法有效统一理解与生成任务，更遑论开放世界涌现能力。我们认为，突破当前困境的关键在于让文本与图像能够在统一的自回归Transformer中被互换表示与处理。为此，我们提出SEED——一种精心设计的图像分词器（tokenizer），可同时赋予大语言模型“看见（理解）”与“绘制（生成）”的能力。我们确立了两大核心设计原则：（1）图像token应独立于二维物理块位置，转而通过一维因果依赖生成，具备与大语言模型中从左到右自回归预测机制相契合的内在 interdependence（相互依赖关系）。（2）图像token应捕捉与词语语义抽象程度一致的高层语义，且在分词器训练阶段需同时针对判别性与重构性进行优化。借助SEED token，大语言模型能够在其原有训练范式（即下一词预测）下执行可扩展的多模态自回归任务。我们通过在交错的文本与视觉数据上进行大规模预训练与指令微调，构建出SEED-LLaMA，其在广泛的多模态理解与生成任务上均展现出优异性能。更重要的是，SEED-LLaMA已展现出组合式涌现能力，例如多轮上下文多模态生成，可像你的AI助手一样运作。
英文摘要：
The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce SEED, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
