arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.03305cs.LGcs.AIcs.CV
收藏

扩散模型是否学习到语义上有意义且高效的表示?

Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?

MIT(麻省理工学院)

Qiyao Liang, Ziming Liu, Ila Fiete

AI总结:

本文通过条件 DDPM 生成二维高斯凸起的受控实验,发现语义有意义的潜在表示是高性能关键,并揭示扩散模型难以因子化学习独立特征,指出需引入归纳偏置以提升数据效率。

摘要:

扩散模型能够完成令人印象深刻的图像生成壮举,包括不寻常的并置,例如宇航员在月球上骑马且阴影位置正确。这些输出表明其具备组合泛化的能力,但模型是如何做到这一点的?我们对条件 DDPM 进行了受控实验,让其学习生成以指定 $x$ 和 $y$ 位置为中心的二维球形高斯凸起。结果表明,语义上有意义的潜在表示的出现是实现高性能的关键。在成功学习的过程中,模型经历了三个不同的潜在表示阶段:(阶段 A)没有潜在结构,(阶段 B)无序状态的二维流形,以及(阶段 C)有序的二维流形。与这些阶段相对应,我们识别出性质不同的生成行为:1)生成多个凸起,2)生成一个凸起但 $x$ 和 $y$ 位置不准确,3)在正确的 $x$ 和 $y$ 位置生成一个凸起。此外,我们表明,即使在特征($x$ 位置与 $y$ 位置)以倾斜频率表示的不平衡数据集下,$x$ 和 $y$ 的学习过程也是耦合的而非因子化的,证明简单的 vanilla 风格扩散模型无法学习到将 $x$ 和 $y$ 的定位因子化为独立一维任务的高效表示。这些发现表明,未来工作需要寻找归纳偏置,以推动生成模型发现并利用输入中可因子化的独立结构,这将是使这些模型进入更高数据效率状态所必需的。

英文摘要:

Diffusion models are capable of impressive feats of image generation with uncommon juxtapositions such as astronauts riding horses on the moon with properly placed shadows. These outputs indicate the ability to perform compositional generalization, but how do the models do so? We perform controlled experiments on conditional DDPMs learning to generate 2D spherical Gaussian bumps centered at specified $x$- and $y$-positions. Our results show that the emergence of semantically meaningful latent representations is key to achieving high performance. En route to successful performance over learning, the model traverses three distinct phases of latent representations: (phase A) no latent structure, (phase B) a 2D manifold of disordered states, and (phase C) a 2D ordered manifold. Corresponding to each of these phases, we identify qualitatively different generation behaviors: 1) multiple bumps are generated, 2) one bump is generated but at inaccurate $x$ and $y$ locations, 3) a bump is generated at the correct $x$ and y location. Furthermore, we show that even under imbalanced datasets where features ($x$- versus $y$-positions) are represented with skewed frequencies, the learning process for $x$ and $y$ is coupled rather than factorized, demonstrating that simple vanilla-flavored diffusion models cannot learn efficient representations in which localization in $x$ and $y$ are factorized into separate 1D tasks. These findings suggest the need for future work to find inductive biases that will push generative models to discover and exploit factorizable independent structures in their inputs, which will be required to vault these models into more data-efficient regimes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑