向 token 分词器与自回归模型对齐的序列建模
Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model
AI总结：
AliTok通过双向编码器与因果解码器设计，解决图像生成中分词器与自回归模型的对齐问题，实现高保真度和可预测性，模型在ImageNet-256和ImageNet-512上取得SOTA结果。
摘要：
自回归图像生成旨在基于先前的token预测下一个token。然而，这个过程受到传统图像token化中固有的双向依赖性挑战，这导致了与自回归模型单向性质的根本不匹配。为此，我们引入了AliTok，一种新的对齐分词器，它改变了token序列的依赖结构。AliTok采用了一个双向编码器，受因果解码器的限制，这种设计迫使编码器生成具有语义丰富性和前向依赖性的token序列。此外，通过引入前缀token，并采用两阶段的分词器训练过程来增强重建性能，AliTok实现了高保真度和可预测性。基于AliTok，一个仅具有17700万参数的标准解码器-only自回归模型在ImageNet-256上实现了gFID为1.44和IS为319.5。扩展到66200万参数，我们的模型达到了gFID为1.28，超过了SOTA扩散方法，采样速度快10倍。在ImageNet-512上，我们的31800万参数模型也实现了SOTA gFID为1.39。代码和权重见https://github.com/ali-vilab/alitok。
英文摘要：
Autoregressive image generation aims to predict the next token based on previous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental misalignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency structure of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorporating prefix tokens and employing a two-stage tokenizer training process to enhance reconstruction performance, AliTok achieves high fidelity and predictability simultaneously. Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on ImageNet-256. Scaling to 662M, our model reaches a gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves a SOTA gFID of 1.39. Code and weights at https://github.com/ali-vilab/alitok.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
