AdaNAT: 探索基于Token的图像生成的自适应策略
AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation
AI总结：
针对非自回归Transformer图像生成依赖人工设计启发式策略、适配性差的问题，提出可学习的AdaNAT方法，将策略确定建模为马尔可夫决策过程，结合对抗奖励通过强化学习训练轻量策略网络，在多个基准数据集上验证了有效性。
摘要：
近期研究已证实基于token的方法在视觉内容生成中的有效性。作为代表性工作，非自回归Transformer（NAT）能够在少量步骤内合成质量不错的图像。然而，NAT通常需要配置包含多个人工设计调度规则的复杂生成策略。这些启发式驱动的规则容易出现次优情况，且需要专业知识与大量人力投入。此外，其“一刀切”的特性无法灵活适配每个样本的不同特征。为解决这些问题，我们提出AdaNAT，这是一种可学习的方法，能自动为每个待生成样本配置适配的策略。具体而言，我们将生成策略的确定过程建模为马尔可夫决策过程。在该框架下，可通过强化学习学习一个轻量级的生成策略网络。重要的是，我们发现简单的奖励设计（如FID或预训练奖励模型）可能无法可靠保证生成样本的预期质量或多样性。因此，我们提出一种对抗式奖励设计来有效指导策略网络的训练。我们在四个基准数据集（即ImageNet-256与512、MS-COCO和CC3M）上开展了全面实验，验证了AdaNAT的有效性。代码与预训练模型将发布于https://github.com/LeapLabTHU/AdaNAT。
英文摘要：
Recent studies have demonstrated the effectiveness of token-based methods for visual content generation. As a representative work, non-autoregressive Transformers (NATs) are able to synthesize images with decent quality in a small number of steps. However, NATs usually necessitate configuring a complicated generation policy comprising multiple manually-designed scheduling rules. These heuristic-driven rules are prone to sub-optimality and come with the requirements of expert knowledge and labor-intensive efforts. Moreover, their one-size-fits-all nature cannot flexibly adapt to the diverse characteristics of each individual sample. To address these issues, we propose AdaNAT, a learnable approach that automatically configures a suitable policy tailored for every sample to be generated. In specific, we formulate the determination of generation policies as a Markov decision process. Under this framework, a lightweight policy network for generation can be learned via reinforcement learning. Importantly, we demonstrate that simple reward designs such as FID or pre-trained reward models, may not reliably guarantee the desired quality or diversity of generated samples. Therefore, we propose an adversarial reward design to guide the training of policy networks effectively. Comprehensive experiments on four benchmark datasets, i.e., ImageNet-256 & 512, MS-COCO, and CC3M, validate the effectiveness of AdaNAT. Code and pre-trained models will be released at https://github.com/LeapLabTHU/AdaNAT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
