The vast applications of deep generative models are anchored in three core capabilities -- generating new instances, reconstructing inputs, and learning compact representations -- across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce Generalized Encoding-Decoding Diffusion Probabilistic Models (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models.
NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Hong Kong（香港大学）；Shenzhen Loop Area Institute（深圳河套学院）；Tsinghua University（清华大学）；Chinese University of Hong Kong（香港中文大学）
As deep Variational Auto-Encoder (VAE) frameworks become more widely used for modeling biomolecular simulation data, we emphasize the capability of the VAE architecture to concurrently maximize the timescale of the latent space while inferring a reduced coordinate, which assists in finding slow processes as according to the variational approach to conformational dynamics. We additionally provide evidence that the VDE framework (Hernández et al., 2017), which uses this autocorrelation loss along with a time-lagged reconstruction loss, obtains a variationally optimized latent coordinate in compa…
We present an approach to synthesizing new graph structures from empirically specified distributions. The generative model is an auto-decoder that learns to synthesize graphs from latent codes. The graph synthesis model is learned jointly with an empirical distribution over the latent codes. Graphs are synthesized using self-attention modules that are trained to identify likely connectivity patterns. Graph-based normalizing flows are used to sample latent codes from the distribution learned by the auto-decoder. The resulting model combines accuracy and scalability. On benchmark datasets of lar…
Latent Variable Double Gaussian Process Model for Decoding Complex Neural Data
Isfahan University of Technology（伊斯法罕理工大学）；Worcester Polytechnique Institute（伍斯特理工学院）；University of Minnesota（明尼苏达大学）；Minneapolis Veterans Affairs Health Care System（明尼阿波利斯退伍军人事务医疗系统）
本文提出一种基于双 GP 与低维潜变量的神经解码器，通过推断神经数据的底层流形来预测标签，并在言语记忆数据集上显著超越现有最先进模型。
Latent variable conditional models, including the latent conditional random fields as a special case, are popular models for many natural language processing and vision processing tasks. The computational complexity of the exact decoding/inference in latent conditional random fields is unclear. In this paper, we try to clarify the computational complexity of the exact decoding. We analyze the complexity and demonstrate that it is an NP-hard problem even on a sequential labeling setting. Furthermore, we propose the latent-dynamic inference (LDI-Naive) method and its bounded version (LDI-Bounded…
Nonlocal operator learning for fMRI encoding and decoding tasks
Department of Computer Science, Idaho State University（计算机科学系，爱达荷州立大学）；Department of Mathematics and Statistics, Idaho State University（数学与统计学系，爱达荷州立大学）
Cross-Session Decoding of Neural Spiking Data via Task-Conditioned Latent Alignment
Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；Departments of Ophthalmology and Bioengineering, University of Pittsburgh（眼科学与生物工程系，匹兹堡大学）；Inria, CEA, Université Paris-Saclay（法国国家科学研究中心（Inria）、法国国家科学研究中心（CEA）、巴黎萨克雷大学）
EEG-VID: Task-Guided Latent Predictive Pretraining for EEG Decoding and Assistive Target Selection
School of Information and Control Engineering, China University of Mining and Technology（中国矿业大学信息与控制工程学院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）；Shanghai Innovation Institute（上海创新研究院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Sabancı University（萨班哲大学）；University of Freiburg（弗赖堡大学）
Causal terminology is often introduced in the interpretation of encoding and decoding models trained on neuroimaging data. In this article, we investigate which causal statements are warranted and which ones are not supported by empirical evidence. We argue that the distinction between encoding and decoding models is not sufficient for this purpose: relevant features in encoding and decoding models carry a different meaning in stimulus- and in response-based experimental paradigms. We show that only encoding models in the stimulus-based setting support unambiguous causal interpretations. By co…
Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and the Transformer attain state-of-the-art results on many tasks. However, they are difficult to parallelize and are thus slow at processing long sequences. RNNs lack parallelism both during training and decoding, while architectures like WaveNet and Transformer are much more parallelizable during training, yet still operate sequentially during decoding. Inspired by [arxiv:1711.00937], we present a method to extend sequence models using discrete latent variables that makes decoding much more parallelizable. We…
Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation
University of California at Los Angeles（加利福尼亚大学洛杉矶分校）；NASA Jet Propulsion Laboratory（美国国家航空航天局喷气推进实验室）
Centre for Research & Technology Hellas（色雷斯研究与技术中心）；Information Technologies Institute（信息技术研究所）
A series of 2D (and 3D) keypoint estimation tasks are built upon heatmap coordinate representation, i.e. a probability map that allows for learnable and spatially aware encoding and decoding of keypoint coordinates on grids, even allowing for sub-pixel coordinate accuracy. In this report, we aim to reproduce the findings of DARK that investigated the 2D heatmap representation by highlighting the importance of the encoding of the ground truth heatmap and the decoding of the predicted heatmap to keypoint coordinates. The authors claim that a) a more principled distribution-aware coordinate decod…
Despite the common usage of a canonical, data-independent, hemodynamic response function (HRF), it is known that the shape of the HRF varies across brain regions and subjects. This suggests that a data-driven estimation of this function could lead to more statistical power when modeling BOLD fMRI data. However, unconstrained estimation of the HRF can yield highly unstable results when the number of free parameters is large. We develop a method for the joint estimation of activation and HRF using a rank constraint causing the estimated HRF to be equal across events/conditions, yet permitting it…
It is well established that temporal organization is critical to memory, and that the ability to temporally organize information is fundamental to many perceptual, cognitive, and motor processes. While our understanding of how the brain processes the spatial context of memories has advanced considerably, our understanding of their temporal organization lags far behind. In this paper, we propose a new approach for elucidating the neural basis of complex behaviors and temporal organization of memories. More specifically, we focus on neural decoding - the prediction of behavioral or experimental …
Neural retrieval models have superseded classic bag-of-words methods such as BM25 as the retrieval framework of choice. However, neural systems lack the interpretability of bag-of-words models; it is not trivial to connect a query change to a change in the latent space that ultimately determines the retrieval results. To shed light on this embedding space, we learn a "query decoder" that, given a latent representation of a neural search engine, generates the corresponding query. We show that it is possible to decode a meaningful query from its latent representation and, when moving in the righ…
INRIA Saclay-Île-de-France（法国国家信息与自动化研究所 萨克莱-法兰西岛分院）；CEA（法国原子能和替代能源委员会）；INRIA Paris - Rocquencourt（法国国家信息与自动化研究所 巴黎-罗克库尔分院）；Institut Mines-Télécom（法国国立矿业电信学院）；Télécom ParisTech（巴黎高等电信学院）；CNRS（法国国家科学研究中心）
Extracting activation patterns from functional Magnetic Resonance Images (fMRI) datasets remains challenging in rapid-event designs due to the inherent delay of blood oxygen level-dependent (BOLD) signal. The general linear model (GLM) allows to estimate the activation from a design matrix and a fixed hemodynamic response function (HRF). However, the HRF is known to vary substantially between subjects and brain regions. In this paper, we propose a model for jointly estimating the hemodynamic response function (HRF) and the activation patterns via a low-rank representation of task effects.This …
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research, New York（微软研究院（纽约））；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research, Redmond（微软研究院（雷德蒙德））
We study the exploration problem in episodic MDPs with rich observations generated from a small number of latent states. Under certain identifiability assumptions, we demonstrate how to estimate a mapping from the observations to latent states inductively through a sequence of regression and clustering steps -- where previously decoded latent states provide labels for later regression problems -- and use it to construct good exploration policies. We provide finite-sample guarantees on the quality of the learned state decoding function and exploration policies, and complement our theory with an…
Cluster of Excellence – Machine Learning for Science（卓越集群——科学机器学习）
A critical challenge for any intelligent system is to infer structure from continuous data streams. Theories of event-predictive cognition suggest that the brain segments sensorimotor information into compact event encodings, which are used to anticipate and interpret environmental dynamics. Here, we introduce a SUrprise-GAted Recurrent neural network (SUGAR) using a novel form of counterfactual regularization. We test the model on a hierarchical sequence prediction task, where sequences are generated by alternating hidden graph structures. Our model learns to both compress the temporal dynami…
SCONE: A Practical, Constraint-Aware Plug-in for Latent Encoding in Learned DNA Storage
Department of Computer Science and Engineering（计算机科学与工程系）；College of Software（软件学院）；Department of Genetics（遗传学系）；Stanford School of Medicine（斯坦福医学院）；Futurewei Technologies Inc.（未来科技有限公司）
SPECTRE: Spectral Pre-training Embeddings with Cylindrical Temporal Rotary Position Encoding for Fine-Grained sEMG-Based Movement Decoding
Clinical Hospital of Chengdu Brain Science Institute（成都脑科学研究院临床医院）；MOE Key Lab for Neuroinformation（教育部神经信息关键实验室）；University of Electronic Science and Technology of China（电子科技大学）；School of Life Science and Technology（生命科学与技术学院）；Center for Information in Medicine（医学信息中心）；Research Unit of NeuroInformation（神经信息研究单位）；Chinese Academy of Medical Sciences（中国医学科学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Faculty of Science and Technology（科学与技术学院）
On finite-dimensional encoding/decoding theorems for neural operators
Departamento de Física, Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学物理系）；Departamento de Matemática, Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学数学系）；Department of Mathematics and Statistics, University of Ottawa（渥太华大学数学与统计学系）
Centre for Artificial Intelligence (CAI), University of Technology Sydney（悉尼科技大学人工智能中心（CAI））
Multi-view alignment, achieving one-to-one correspondence of multi-view inputs, is critical in many real-world multi-view applications, especially for cross-view data analysis problems. Recently, an increasing number of works study this alignment problem with Canonical Correlation Analysis (CCA). However, existing CCA models are prone to misalign the multiple views due to either the neglect of uncertainty or the inconsistent encoding of the multiple views. To tackle these two issues, this paper studies multi-view alignment from the Bayesian perspective. Delving into the impairments of inconsis…
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Pittsburgh（匹兹堡大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
We investigate the problems of model estimation and reward-free learning in episodic Block MDPs. In these MDPs, the decision maker has access to rich observations or contexts generated from a small number of latent states. We are first interested in estimating the latent state decoding function (the mapping from the observations to latent states) based on data generated under a fixed behavior policy. We derive an information-theoretical lower bound on the error rate for estimating this function and present an algorithm approaching this fundamental limit. In turn, our algorithm also provides es…
Task-Aware Calibration: Provably Optimal Decoding in LLMs
School of Computation, Information & Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Amsterdam（阿姆斯特丹大学）
Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学北京校区人工智能学院）；MiLM Plus, Xiaomi Inc., Beijing, China（小米公司北京MiLM Plus团队）；University of Modena and Reggio Emilia（莫德纳和雷吉奥艾米莉亚大学）；University of Pisa, Italy（比萨大学）
Harbin Institute of Technology at Shenzhen（深圳哈尔滨工业大学）；The Chinese University of Hong Kong at Shenzhen（香港中文大学（深圳））
The crux of single-channel speech separation is how to encode the mixture of signals into such a latent embedding space that the signals from different speakers can be precisely separated. Existing methods for speech separation either transform the speech signals into frequency domain to perform separation or seek to learn a separable embedding space by constructing a latent domain based on convolutional filters. While the latter type of methods learning an embedding space achieves substantial improvement for speech separation, we argue that the embedding space defined by only one latent domai…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；National University of Sciences and Technology（国立科技大学）
Understanding E3 ligase and target substrate interactions are important for cell biology and therapeutic development. However, experimental identification of E3 target relationships is not an easy task due to the labor-intensive nature of the experiments. In this article, a sequence-based E3-target prediction model is proposed for the first time. The proposed framework utilizes composition of k-spaced amino acid pairs (CKSAAP) to learn the relationship between E3 ligases and their target protein. A class separable latent space encoding scheme is also devised that provides a compressed represen…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Bernstein Center Freiburg, University of Freiburg（弗赖堡大学伯恩斯坦弗莱堡中心）
Pattern recognition in neuroimaging distinguishes between two types of models: encoding- and decoding models. This distinction is based on the insight that brain state features, that are found to be relevant in an experimental paradigm, carry a different meaning in encoding- than in decoding models. In this paper, we argue that this distinction is not sufficient: Relevant features in encoding- and decoding models carry a different meaning depending on whether they represent causal- or anti-causal relations. We provide a theoretical justification for this argument and conclude that causal infer…
