现代神经科学数据集使得人们能够对复杂行为与许多同时记录的神经元活动之间的关系展开前所未有的探究。尽管潜在变量模型能够成功地从此类记录中提取低维嵌入，但利用它们生成逼真的脉冲数据，尤其是以依赖于行为的方式生成，仍然构成挑战。本文提出用于神经脉冲数据的潜在扩散模型（Latent Diffusion for Neural Spiking data，LDNS），这是一种具有低维潜在空间的基于扩散的生成模型：LDNS采用带有结构化状态空间（S4）层的自编码器，将离散的高维脉冲数据投影到连续的、时间对齐的潜在变量中。在这些推断出的潜在变量上，我们训练具有强表达能力的（条件）扩散模型，从而能够采样具有逼真单神经元和群体脉冲统计特征的神经活动。我们在合成数据上验证LDNS，其能够准确恢复潜在结构、放电率和脉冲统计特征。接下来，我们通过生成模拟人类尝试言语期间皮层活动的可变长度数据，证明了该模型的灵活性。我们展示了如何为LDNS配备一个具有强表达能力的观测模型，该模型可解释并非由潜在状态介导的单神经元动态，从而进一步提升生成样本的逼真度。最后，在多种伸手行为期间的运动皮层活动上训练的条件LDNS，能够在给定伸手方向或未见过的伸手轨迹时生成逼真的脉冲数据。总之，LDNS同时实现了低维潜在变量推断和神经脉冲数据集的逼真条件生成，为模拟可通过实验检验的假说开辟了更多可能性。
英文摘要：
Modern datasets in neuroscience enable unprecedented inquiries into the relationship between complex behaviors and the activity of many simultaneously recorded neurons. While latent variable models can successfully extract low-dimensional embeddings from such recordings, using them to generate realistic spiking data, especially in a behavior-dependent manner, still poses a challenge. Here, we present Latent Diffusion for Neural Spiking data (LDNS), a diffusion-based generative model with a low-dimensional latent space: LDNS employs an autoencoder with structured state-space (S4) layers to project discrete high-dimensional spiking data into continuous time-aligned latents. On these inferred latents, we train expressive (conditional) diffusion models, enabling us to sample neural activity with realistic single-neuron and population spiking statistics. We validate LDNS on synthetic data, accurately recovering latent structure, firing rates, and spiking statistics. Next, we demonstrate its flexibility by generating variable-length data that mimics human cortical activity during attempted speech. We show how to equip LDNS with an expressive observation model that accounts for single-neuron dynamics not mediated by the latent state, further increasing the realism of generated samples. Finally, conditional LDNS trained on motor cortical activity during diverse reaching behaviors can generate realistic spiking data given reach direction or unseen reach trajectories. In summary, LDNS simultaneously enables inference of low-dimensional latents and realistic conditional generation of neural spiking datasets, opening up further possibilities for simulating experimentally testable hypotheses.
Spiking neural networks (SNNs) can be run on neuromorphic devices with ultra-high speed and ultra-low energy consumption because of their binary and event-driven nature. Therefore, SNNs are expected to have various applications, including as generative models being running on edge devices to create high-quality images. In this study, we build a variational autoencoder (VAE) with SNN to enable image generation. VAE is known for its stability among generative models; recently, its quality advanced. In vanilla VAE, the latent space is represented as a normal distribution, and floating-point calcu…
Inria（法国国家信息与自动化技术研究院）；PSL Research University（巴黎社会科学高等研究院）；CMAP（巴黎高等理工学院应用数学与计算科学实验室）；Ecole Polytechnique Palaiseau, France（法国巴黎高等理工学院Palaiseau分校）；Sakana AI Tokyo, Japan（日本东京Sakana AI公司）
Diffusion models have achieved unprecedented performance in generative modeling. The commonly-adopted formulation of the latent code of diffusion models is a sequence of gradually denoised samples, as opposed to the simpler (e.g., Gaussian) latent space of GANs, VAEs, and normalizing flows. This paper provides an alternative, Gaussian formulation of the latent space of various diffusion models, as well as an invertible DPM-Encoder that maps images into the latent space. While our formulation is purely based on the definition of diffusion models, we demonstrate several intriguing consequences. …
Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation
Polymathic AI；Flatiron Institute；University of Liège（列日大学）；New York University（纽约大学）；Princeton University（普林斯顿大学）；Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM（巴黎-萨克莱大学、巴黎-cite大学、CEA、CNRS、AIM）
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
UCLA（加州大学洛杉矶分校）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Salesforce Research（赛富时研究院）；Google Brain（谷歌大脑）；Peking University（北京大学）；Tsinghua University（清华大学）
Latent Schr{ö}dinger Bridge Diffusion Model for Generative Learning
School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）；Center of Statistical Research and School of Statistics, Southwestern University of Finance and Economics（西南财经大学统计研究中心与统计学院）
Despite the remarkable progress in deep generative models, synthesizing high-resolution and temporally coherent videos still remains a challenge due to their high-dimensionality and complex temporal dynamics along with large spatial variations. Recent works on diffusion models have shown their potential to solve this challenge, yet they suffer from severe computation- and memory-inefficiency that limit the scalability. To handle this issue, we propose a novel generative model for videos, coined projected latent video diffusion models (PVDM), a probabilistic diffusion model which learns a video…
3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation
Artificial Intelligence Research Laboratory（人工智能研究实验室）；College of Information Sciences and Technology（信息科学与技术学院）；Center for Artificial Intelligence Foundations and Scientific Applications（人工智能基础与科学应用中心）；Institute for Computational and Data Sciences（计算与数据科学研究所）；The Pennsylvania State University（宾夕法尼亚州立大学）
VAMP-Diff: VampPrior Latent Diffusion for Photoplethysmography Modeling
Department of Electrical and Computer Engineering（电气与计算机工程系）；Clarkson University（克林顿大学）；Department of Computer Science（计算机科学系）；Worcester Polytechnic Institute（沃思格理工学院）
