arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.13774cs.LG
收藏

从点到函数:扩散模型的无穷维表示

From Points to Functions: Infinite-dimensional Representations in Diffusion Models

Mila(米拉研究所);Université de Montréal(蒙特利尔大学);KTH Stockholm(斯德哥尔摩皇家理工学院);MPI-IS(马克斯·普朗克智能系统研究所);ETH Zurich(苏黎世联邦理工学院)

Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, Arash Mehrjou

摘要:

基于扩散的生成模型学习迭代地将非结构化噪声转移到复杂的目标分布,而不是生成对抗网络(GANs)或变分自编码器(VAE)的解码器,后者在单步的中从目标分布中产生样本。因此,在扩散模型中,每个样本都自然地连接到随机轨迹,该随机轨迹是学习的随机微分方程(RSDs)的解。生成模型只关心这个轨迹的最终状态,它从所需的分布中提供样本。Abstreiter et. Al指出,这些随机轨迹可以被看作是连续的过滤器,一路上冲刷掉信息。因此,有理由询问是否存在中间时间步长,在该中间时间步长处,所保留的信息对于给定的下游任务是最优的。在这项工作中,我们表明,从不同的时间步长的信息内容的组合提供了一个严格更好的表示下游的任务。我们引入了一个注意力和复发为基础的模块,“学会混合”的各种时间步长的信息内容,从而产生的表示导致在下游任务的性能优越。

英文摘要:

Diffusion-based generative models learn to iteratively transfer unstructured noise to a complex target distribution as opposed to Generative Adversarial Networks (GANs) or the decoder of Variational Autoencoders (VAEs) which produce samples from the target distribution in a single step. Thus, in diffusion models every sample is naturally connected to a random trajectory which is a solution to a learned stochastic differential equation (SDE). Generative models are only concerned with the final state of this trajectory that delivers samples from the desired distribution. Abstreiter et. al showed that these stochastic trajectories can be seen as continuous filters that wash out information along the way. Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task. In this work, we show that a combination of information content from different time steps gives a strictly better representation for the downstream task. We introduce an attention and recurrence based modules that ``learn to mix'' information content of various time-steps such that the resultant representation leads to superior performance in downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑