NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Hong Kong（香港大学）；Shenzhen Loop Area Institute（深圳河套学院）；Tsinghua University（清华大学）；Chinese University of Hong Kong（香港中文大学）
Weijian Mai, Mu Nan, Yu Zhu, Jiahang Cao, Rui Zhang, Yuqin Dai, Chunfeng Song, Andrew F. Luo, Jiamin Wu
Visual encoding and decoding models act as gateways to understanding the neural mechanisms underlying human visual perception. Typically, visual encoding models that predict brain activity from stimuli and decoding models that reproduce stimuli from brain activity are treated as distinct tasks, requiring separate models and training procedures. This separation is inefficient and fails to model the consistency between encoding and decoding processes. To address this limitation, we propose NeuroFlow, the first unified framework that jointly models visual encoding and decoding from neural activity within a single flow model. NeuroFlow introduces two key components: (1) NeuroVAE is designed as a variational backbone to model neural variability and establish a compact, semantically structured latent space for bidirectional modeling across visual and neural modalities. (2) Cross-modal Flow Matching (XFM) bypasses the typical paradigm of noise-to-data diffusion guided by a specific modality condition, instead learning a reversibly consistent flow model between visual and neural latent distributions. For the first time, visual encoding and decoding are reformulated as a time-dependent, reversible process within a shared latent space for unified modeling. Empirical results demonstrate that NeuroFlow achieves superior overall performance in visual encoding and decoding tasks with higher computational efficiency compared to any isolated methods. We further analyze principal factors that steer the model toward encoding-decoding consistency and, through brain functional analyses, demonstrate that NeuroFlow captures consistent activation patterns underlying neural variability. NeuroFlow marks a major step toward unified visual encoding and decoding from neural activity, providing mechanistic insights that inform future bidirectional visual brain-computer interfaces.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Sabancı University（萨班哲大学）；University of Freiburg（弗赖堡大学）
Causal terminology is often introduced in the interpretation of encoding and decoding models trained on neuroimaging data. In this article, we investigate which causal statements are warranted and which ones are not supported by empirical evidence. We argue that the distinction between encoding and decoding models is not sufficient for this purpose: relevant features in encoding and decoding models carry a different meaning in stimulus- and in response-based experimental paradigms. We show that only encoding models in the stimulus-based setting support unambiguous causal interpretations. By co…
Despite the common usage of a canonical, data-independent, hemodynamic response function (HRF), it is known that the shape of the HRF varies across brain regions and subjects. This suggests that a data-driven estimation of this function could lead to more statistical power when modeling BOLD fMRI data. However, unconstrained estimation of the HRF can yield highly unstable results when the number of free parameters is large. We develop a method for the joint estimation of activation and HRF using a rank constraint causing the estimated HRF to be equal across events/conditions, yet permitting it…
INRIA Saclay-Île-de-France（法国国家信息与自动化研究所 萨克莱-法兰西岛分院）；CEA（法国原子能和替代能源委员会）；INRIA Paris - Rocquencourt（法国国家信息与自动化研究所 巴黎-罗克库尔分院）；Institut Mines-Télécom（法国国立矿业电信学院）；Télécom ParisTech（巴黎高等电信学院）；CNRS（法国国家科学研究中心）
Extracting activation patterns from functional Magnetic Resonance Images (fMRI) datasets remains challenging in rapid-event designs due to the inherent delay of blood oxygen level-dependent (BOLD) signal. The general linear model (GLM) allows to estimate the activation from a design matrix and a fixed hemodynamic response function (HRF). However, the HRF is known to vary substantially between subjects and brain regions. In this paper, we propose a model for jointly estimating the hemodynamic response function (HRF) and the activation patterns via a low-rank representation of task effects.This …
Nonlocal operator learning for fMRI encoding and decoding tasks
Department of Computer Science, Idaho State University（计算机科学系，爱达荷州立大学）；Department of Mathematics and Statistics, Idaho State University（数学与统计学系，爱达荷州立大学）
Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation
University of California at Los Angeles（加利福尼亚大学洛杉矶分校）；NASA Jet Propulsion Laboratory（美国国家航空航天局喷气推进实验室）
On finite-dimensional encoding/decoding theorems for neural operators
Departamento de Física, Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学物理系）；Departamento de Matemática, Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学数学系）；Department of Mathematics and Statistics, University of Ottawa（渥太华大学数学与统计学系）
Centre for Research & Technology Hellas（色雷斯研究与技术中心）；Information Technologies Institute（信息技术研究所）
A series of 2D (and 3D) keypoint estimation tasks are built upon heatmap coordinate representation, i.e. a probability map that allows for learnable and spatially aware encoding and decoding of keypoint coordinates on grids, even allowing for sub-pixel coordinate accuracy. In this report, we aim to reproduce the findings of DARK that investigated the 2D heatmap representation by highlighting the importance of the encoding of the ground truth heatmap and the decoding of the predicted heatmap to keypoint coordinates. The authors claim that a) a more principled distribution-aware coordinate decod…
SPECTRE: Spectral Pre-training Embeddings with Cylindrical Temporal Rotary Position Encoding for Fine-Grained sEMG-Based Movement Decoding
Clinical Hospital of Chengdu Brain Science Institute（成都脑科学研究院临床医院）；MOE Key Lab for Neuroinformation（教育部神经信息关键实验室）；University of Electronic Science and Technology of China（电子科技大学）；School of Life Science and Technology（生命科学与技术学院）；Center for Information in Medicine（医学信息中心）；Research Unit of NeuroInformation（神经信息研究单位）；Chinese Academy of Medical Sciences（中国医学科学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Faculty of Science and Technology（科学与技术学院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Bernstein Center Freiburg, University of Freiburg（弗赖堡大学伯恩斯坦弗莱堡中心）
Pattern recognition in neuroimaging distinguishes between two types of models: encoding- and decoding models. This distinction is based on the insight that brain state features, that are found to be relevant in an experimental paradigm, carry a different meaning in encoding- than in decoding models. In this paper, we argue that this distinction is not sufficient: Relevant features in encoding- and decoding models carry a different meaning depending on whether they represent causal- or anti-causal relations. We provide a theoretical justification for this argument and conclude that causal infer…
Brain networks in fMRI are typically identified using spatial independent component analysis (ICA), yet mathematical constraints such as sparse coding and positivity both provide alternate biologically-plausible frameworks for generating brain networks. Non-negative Matrix Factorization (NMF) would suppress negative BOLD signal by enforcing positivity. Spatial sparse coding algorithms ($L1$ Regularized Learning and K-SVD) would impose local specialization and a discouragement of multitasking, where the total observed activity in a single voxel originates from a restricted number of possible br…
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
Neural encoding and decoding, which aim to characterize the relationship between stimuli and brain activities, have emerged as an important area in cognitive neuroscience. Traditional encoding models, which focus on feature extraction and mapping, consider the brain as an input-output mapper without inner states. In this work, inspired by the fact that human brain acts like a state machine, we proposed a novel encoding framework that combines information from both the external world and the inner state to predict brain activity. The framework comprises two parts: forward encoding model that de…
University of Surrey（萨里大学）；University of Applied Sciences Stralsund（施特拉尔松德应用技术大学）
Experimental studies support the notion of spike-based neuronal information processing in the brain, with neural circuits exhibiting a wide range of temporally-based coding strategies to rapidly and efficiently represent sensory stimuli. Accordingly, it would be desirable to apply spike-based computation to tackling real-world challenges, and in particular transferring such theory to neuromorphic systems for low-power embedded applications. Motivated by this, we propose a new supervised learning method that can train multilayer spiking neural networks to solve classification problems based on …
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Inria Bordeaux（法国国家信息与自动化研究所波尔多分所）；LaBRI, Université de Bordeaux（波尔多大学拉布里实验室）；Université de Bordeaux（波尔多大学）；TU Berlin（柏林工业大学）；National University of Singapore（新加坡国立大学）；Microsoft（微软公司）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术研究所）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）
Mila（米拉）；Harvard University（哈佛大学）；Panasonic（松下）；HEC Montréal（蒙特利尔高等商学院）；Université de Montréal（蒙特利尔大学）
Generative models have achieved impressive results in many domains including image and text generation. In the natural sciences, generative models have led to rapid progress in automated drug discovery. Many of the current methods focus on either 1-D or 2-D representations of typically small, drug-like molecules. However, many molecules require 3-D descriptors and exceed the chemical complexity of commonly used dataset. We present a method to encode and decode the position of atoms in 3-D molecules from a dataset of nearly 50,000 stable crystal unit cells that vary from containing 1 to over 10…
State Key Laboratory of Software Development Environment（软件开发环境国家重点实验室）；School of Computer Science, Beihang University（北京航空航天大学计算机学院）
In the present paper, we propose the model of {\it structural information learning machines} (SiLeM for short), leading to a mathematical definition of learning by merging the theories of computation and information. Our model shows that the essence of learning is {\it to gain information}, that to gain information is {\it to eliminate uncertainty} embedded in a data space, and that to eliminate uncertainty of a data space can be reduced to an optimization problem, that is, an {\it information optimization problem}, which can be realized by a general {\it encoding tree method}. The principle a…
