arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.15098cs.CVcs.AIcs.LG
收藏

OminiControl:扩散Transformer的最小化通用控制

OminiControl: Minimal and Universal Control for Diffusion Transformer

National University of Singapore(新加坡国立大学)

Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, Xinchao Wang

AI总结:

针对现有图像条件方法参数开销大或任务通用性差的问题,提出OminiControl,通过最小化架构设计、统一序列处理和动态位置编码实现仅0.1%额外参数的通用控制,性能超越专用方法并构建Subjects200K数据集。

摘要:

我们提出OminiControl,这是一种重新思考图像条件如何集成到扩散Transformer(DiT)架构中的新方法。当前的图像条件方法要么引入大量参数开销,要么仅能有效处理特定控制任务,限制了其实用性和通用性。OminiControl通过三项关键创新解决这些局限:(1)最小化架构设计,利用DiT自身的VAE编码器和Transformer块,仅需0.1%的额外参数;(2)统一序列处理策略,将条件token与图像token结合以实现灵活的token交互;(3)动态位置编码机制,适应空间对齐和非对齐控制任务。我们的大量实验表明,这种精简方法不仅匹配,甚至超过了专用方法在多个条件任务中的性能。为克服主体驱动生成中的数据限制,我们还引入Subjects200K,这是一个使用DiT模型自身合成的大规模身份一致图像对数据集。这项工作证明,无需架构复杂性即可实现有效的图像控制,为高效且通用的图像生成系统开辟了新可能。

英文摘要:

We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures. Current image conditioning methods either introduce substantial parameter overhead or handle only specific control tasks effectively, limiting their practical versatility. OminiControl addresses these limitations through three key innovations: (1) a minimal architectural design that leverages the DiT's own VAE encoder and transformer blocks, requiring just 0.1% additional parameters; (2) a unified sequence processing strategy that combines condition tokens with image tokens for flexible token interactions; and (3) a dynamic position encoding mechanism that adapts to both spatially-aligned and non-aligned control tasks. Our extensive experiments show that this streamlined approach not only matches but surpasses the performance of specialized methods across multiple conditioning tasks. To overcome data limitations in subject-driven generation, we also introduce Subjects200K, a large-scale dataset of identity-consistent image pairs synthesized using DiT models themselves. This work demonstrates that effective image control can be achieved without architectural complexity, opening new possibilities for efficient and versatile image generation systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑