arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.04299stat.MLcs.AIcs.CCcs.CLcs.LG
收藏

视觉自回归模型的电路复杂度界

Circuit Complexity Bounds for Visual Autoregressive Model

The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing at UC Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)

Yekun Ke, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song

AI总结:

本文研究图像生成中 Visual AutoRegressive(VAR)模型的电路复杂度,证明其可由隐藏维度 d≤O(n)、poly(n) 精度的统一 TC^0 阈值电路模拟,从而首次严格刻画其表达能力边界。

摘要:

理解特定模型的表达能力对于把握其容量局限性至关重要。最近,多项研究已经为 Transformer 架构建立了电路复杂度界。此外,Visual AutoRegressive(VAR,视觉自回归)模型已崛起为图像生成领域的一种重要方法,在生成高质量图像方面优于 Diffusion Transformers 等以往技术。在本研究中,我们考察 VAR 模型的电路复杂度并建立一个界。我们的主要结果表明,VAR 模型等价于由隐藏维度 d ≤ O(n) 且具有 poly(n) 精度的统一 TC^0 阈值电路进行的模拟。这是首个在 VAR 模型表现令人印象深刻的情况下,严格揭示其表达能力局限性的研究。我们相信,我们的发现将为这些模型的内在约束提供宝贵见解,并指导未来开发更高效、更具表达能力的架构。

英文摘要:

Understanding the expressive ability of a specific model is essential for grasping its capacity limitations. Recently, several studies have established circuit complexity bounds for Transformer architecture. Besides, the Visual AutoRegressive (VAR) model has risen to be a prominent method in the field of image generation, outperforming previous techniques, such as Diffusion Transformers, in generating high-quality images. We investigate the circuit complexity of the VAR model and establish a bound in this study. Our primary result demonstrates that the VAR model is equivalent to a simulation by a uniform $\mathsf{TC}^0$ threshold circuit with hidden dimension $d \leq O(n)$ and $\mathrm{poly}(n)$ precision. This is the first study to rigorously highlight the limitations in the expressive power of VAR models despite their impressive performance. We believe our findings will offer valuable insights into the inherent constraints of these models and guide the development of more efficient and expressive architectures in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑