arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04671cs.CV
收藏

CAR:面向视觉生成的可控自回归建模

CAR: Controllable Autoregressive Modeling for Visual Generation

School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Tencent Youtu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Washington(华盛顿大学)

Ziyu Yao, Jialin Li, Yifeng Zhou, Yong Liu, Xi Jiang, Chengjie Wang, Feng Zheng, Yuexian Zou, Lei Li

AI总结:

针对自回归视觉生成模型可控性研究不足的问题,提出即插即用的CAR框架,将条件控制融入多尺度隐变量建模,在少训练资源下实现多条件可控生成且图像质量更优,为首款预训练自回归视觉生成控制框架。

摘要:

可控生成能够对生成输出实现细粒度控制,已成为视觉生成模型领域的核心研究方向。当前视觉生成主要有两类技术路径:扩散模型与自回归模型。以ControlNet、T2I-Adapter为代表的扩散模型具备成熟的控制机制,而自回归模型尽管生成质量出色、可扩展性强,其可控性与灵活性方面的研究仍显不足。本研究提出可控自回归建模(Controllable AutoRegressive Modeling,简称CAR),这是一种全新的即插即用框架,将条件控制融入多尺度隐变量建模,可在预训练视觉自回归模型内实现高效的可控生成。CAR逐步优化并捕获控制表征,将其注入预训练模型的每个自回归步骤以引导生成过程。该方法在多种条件类型下均展现出优异的可控性,且图像质量优于现有方法。此外,与预训练模型所需的训练资源相比,CAR仅需极少的训练资源即可实现强大的泛化能力。据我们所知,我们是首个为预训练自回归视觉生成模型提出控制框架的研究。

英文摘要:

Controllable generation, which enables fine-grained control over generated outputs, has emerged as a critical focus in visual generative models. Currently, there are two primary technical approaches in visual generation: diffusion models and autoregressive models. Diffusion models, as exemplified by ControlNet and T2I-Adapter, offer advanced control mechanisms, whereas autoregressive models, despite showcasing impressive generative quality and scalability, remain underexplored in terms of controllability and flexibility. In this study, we introduce Controllable AutoRegressive Modeling (CAR), a novel, plug-and-play framework that integrates conditional control into multi-scale latent variable modeling, enabling efficient control generation within a pre-trained visual autoregressive model. CAR progressively refines and captures control representations, which are injected into each autoregressive step of the pre-trained model to guide the generation process. Our approach demonstrates excellent controllability across various types of conditions and delivers higher image quality compared to previous methods. Additionally, CAR achieves robust generalization with significantly fewer training resources compared to those required for pre-training the model. To the best of our knowledge, we are the first to propose a control framework for pre-trained autoregressive visual generation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑