LINA: 基于连续令牌的线性自回归图像生成模型
LINA: Linear Autoregressive Image Generative Models with Continuous Tokens
AI总结：
LINA是一种基于线性注意力的高效文本到图像生成模型，通过改进归一化和门控机制，在计算效率和生成质量上取得平衡。
摘要：
自回归模型使用连续令牌形成了一种有前景的视觉生成范式，尤其是在文本到图像（T2I）合成中，但它们面临计算成本高的问题。我们研究了如何在该框架中设计计算高效的线性注意力机制。具体而言，我们对不同设计选择下参数数量的缩放行为进行了系统性的实证分析，重点关注（1）线性注意力中的归一化范式（基于除法的 vs. 基于减法的）以及（2）深度卷积用于局部性增强。我们的结果表明，虽然基于减法的归一化在图像分类中有效，但基于除法的归一化在线性生成变压器中缩放更好。此外，将卷积用于局部性建模在自回归生成中起着关键作用，这与扩散模型中的发现一致。我们进一步将常用于因果线性注意力中的门控机制扩展到双向设置，并提出KV门控。通过在键和值状态中引入数据无关的可学习参数，KV门控为每个令牌分配记忆权重，使内存管理具有与语言模型中的遗忘门相似的灵活性。基于这些发现，我们提出了LINA，一种简单且计算高效的T2I模型，完全基于线性注意力，能够从用户指令中生成高保真度的1024x1024图像。LINA在类别条件和T2I基准测试中均取得了竞争性的性能，分别在ImageNet上获得2.18 FID（约1.4B参数）和GenEval上获得0.74（约1.5B参数）。单个线性注意力模块将FLOPs减少约61%相比softmax注意力。代码和模型可在：https://github.com/techmonsterwang/LINA获取。
英文摘要：
Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis, but they suffer from high computational cost. We study how to design compute-efficient linear attention within this framework. Specifically, we conduct a systematic empirical analysis of scaling behavior with respect to parameter counts under different design choices, focusing on (1) normalization paradigms in linear attention (division-based vs. subtraction-based) and (2) depthwise convolution for locality augmentation. Our results show that although subtraction-based normalization is effective for image classification, division-based normalization scales better for linear generative transformers. In addition, incorporating convolution for locality modeling plays a crucial role in autoregressive generation, consistent with findings in diffusion models. We further extend gating mechanisms, commonly used in causal linear attention, to the bidirectional setting and propose a KV gate. By introducing data-independent learnable parameters to the key and value states, the KV gate assigns token-wise memory weights, enabling flexible memory management similar to forget gates in language models. Based on these findings, we present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions. LINA achieves competitive performance on both class-conditional and T2I benchmarks, obtaining 2.18 FID on ImageNet (about 1.4B parameters) and 0.74 on GenEval (about 1.5B parameters). A single linear attention module reduces FLOPs by about 61 percent compared to softmax attention. Code and models are available at: https://github.com/techmonsterwang/LINA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
