Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (Instance-Conditioned GAN) to train another mapping from brain response to fine-grained image features using a variational approach (Cortex2Detail). Image reconstructions obtained by our proposed method achieve state-of-the-art semantic fidelity, while yielding good fine-grained similarity with the ground-truth stimulus. Our code is available at: https://github.com/zijin-gu/meshconv-decoding.git.
A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli
Beijing Institute of Basic Medical Sciences（北京基础医学研究所）；School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)（合肥工业大学计算机与信息工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Zhejiang University（浙江大学）；Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院澳大利亚人工智能研究所）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
University of Rome, Tor Vergata（罗马第二大学）；Istituto di Bioimmagini e Fisiologia Molecolare, CNR（意大利国家研究委员会生物成像与分子生理学研究所）；Martinos Center for Biomedical Imaging（马丁诺斯生物医学成像中心）；MGH（麻省总医院）；Harvard Medical School（哈佛医学院）
Brain decoding is a field of computational neuroscience that uses measurable brain activity to infer mental states or internal representations of perceptual inputs. Therefore, we propose a novel approach to brain decoding that also relies on semantic and contextual similarity. We employ an fMRI dataset of natural image vision and create a deep learning decoding pipeline inspired by the existence of both bottom-up and top-down processes in human vision. We train a linear brain-to-feature model to map fMRI activity features to visual stimuli features, assuming that the brain projects visual info…
Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI
Obvious Research（奥布维研究机构）；GHU Paris Psychiatrie et Neurosciences（巴黎精神病学与神经科学大学医院集团）；Université Paris Cité（巴黎西岱大学）；Centre Imagerie du Nord（北方影像中心）；Raidium（雷迪姆公司）；FrontLab, Sorbonne Université（索邦大学前沿实验室）；Institut du Cerveau（大脑研究所）；Centre de NeuroImagerie de Recherche, Sorbonne Université（索邦大学神经影像研究中心）；Institut des Systèmes Intelligents et de la Robotique, Sorbonne Université（索邦大学智能系统与机器人研究所）
NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction
Tongji University（同济大学）；Ohio State University（俄亥俄州立大学）；University of Technology Sydney（悉尼科技大学）；Chinese Academy of Sciences（中国科学院）；Beijing Anding Hospital（北京安定医院）
Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
NeuralDiffuser: Neuroscience-inspired Diffusion Guidance for fMRI Visual Reconstruction
Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Xi’an Jiaotong University（西安交通大学）；School of Electrical Engineering（电气工程学院）；Xi’an University of Technology（西安理工大学）
PLA strategy support force information engineering university（中国人民解放军战略支援部队信息工程大学）
In the visual decoding domain, visually reconstructing presented images given the corresponding human brain activity monitored by functional magnetic resonance imaging (fMRI) is difficult, especially when reconstructing viewed natural images. Visual reconstruction is a conditional image generation on fMRI data and thus generative adversarial network (GAN) for natural image generation is recently introduced for this task. Although GAN-based methods have greatly improved, the fidelity and naturalness of reconstruction are still unsatisfactory due to the small number of fMRI data samples and the …
CerCo, CNRS（法国国家科学研究中心脑与认知研究中心）；Université de Toulouse（图卢兹大学）
Decoding and reconstructing images from brain imaging data is a research area of high interest. Recent progress in deep generative neural networks has introduced new opportunities to tackle this problem. Here, we employ a recently proposed large-scale bi-directional generative adversarial network, called BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGAN converts images into a 120-dimensional latent space which encodes class and attribute information together, and can also reconstruct images based on their latent vectors. We computed a linear mapping between fMRI d…
Reconstructing perceived natural images from fMRI signals is one of the most engaging topics of neural decoding research. Prior studies had success in reconstructing either the low-level image features or the semantic/high-level aspects, but rarely both. In this study, we utilized an Instance-Conditioned GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate semantic attributes and preserved low-level details. The IC-GAN model takes as input a 119-dim noise vector and a 2048-dim instance feature vector extracted from a target image via a self-supervised learning model (…
National Digital Switching System Engineering and Technological Research Center（国家数字交换系统工程技术研究中心）
In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of the deep network computation. However, comparing with the goal of decoding orientation, position and object category from activities in visual cortex, accurate reconstruction of image stimuli from human fMRI is a still challenging work. In this paper, the c…
Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding
State Key Laboratory of Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Institute of Brain-Machine Interface, Nanjing University（脑机接口研究院，南京大学）
fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding
School of Data Science, Fudan University（复旦大学数据科学学院）；Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）
Deep Neural Encoder-Decoder Model to Relate fMRI Brain Activity with Naturalistic Stimuli
Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne (EPFL), Geneva, Switzerland（Neuro-X研究所，瑞士联邦理工学院（EPFL），日内瓦，瑞士）；Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland（放射学与医学信息学系，日内瓦大学，日内瓦，瑞士）；Swiss Center for Affective Sciences, University of Geneva, Geneva, Switzerland（瑞士情感科学中心，日内瓦大学，日内瓦，瑞士）；Geneva Neuroscience Center, University of Geneva, Geneva, Switzerland（日内瓦神经科学中心，日内瓦大学，日内瓦，瑞士）
VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI
Department of Electrical and Computer Engineering, New York University Tandon School of Engineering（电气工程系，纽约大学塔能工程学院）；Department of Computer Science and Engineering, New York University Tandon School of Engineering（计算机科学与工程系，纽约大学塔能工程学院）
Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applyin…
Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models
Department of Psychiatry and Behavioral Sciences, Stanford University（精神病学与行为科学系，斯坦福大学）；Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）；Department of Neurobiology, University of Chicago（神经生物学系，芝加哥大学）；Google DeepMind（谷歌DeepMind）
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University of Science and Technology（南京理工大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that captures the blood oxygen level in a subject's brain while the subject either rests or performs a variety of functional tasks under different conditions. Given fMRI data, the problem of inferring the task, known as task state decoding, is challenging due to the high dimensionality (hundreds of million sampling points per datum) and complex spatio-temporal blood flow patterns inherent in the data. In this work, we propose to tackle the fMRI task state decoding problem by casting it as a 4D spatio-temporal classificatio…
The dispute of how the human brain represents conceptual knowledge has been argued in many scientific fields. Brain imaging studies have shown that the spatial patterns of neural activation in the brain are correlated with thinking about different semantic categories of words (for example, tools, animals, and buildings) or when viewing the related pictures. In this paper, we present a computational model that learns to predict the neural activation captured in functional magnetic resonance imaging (fMRI) data of test words. Unlike the models with hand-crafted features that have been used in th…
Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding
Gamania Digital Entertainment Co., Ltd.（Gamania数字娱乐有限公司）；Institute of Information Science, Academia Sinica（Academia Sinica信息科学研究所）；Sony Computer Science Laboratories, Inc.（索尼计算机科学实验室有限公司）
NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery
University of Minnesota（明尼苏达大学）；Alljoined；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Stability AI/Medical AI Research Center (MedARC)（Stability AI/医疗AI研究中心（MedARC））
Deep neural networks (DNNs) are being increasingly used to make predictions from functional magnetic resonance imaging (fMRI) data. However, they are widely seen as uninterpretable "black boxes", as it can be difficult to discover what input information is used by the DNN in the process, something important in both cognitive neuroscience and clinical applications. A saliency map is a common approach for producing interpretable visualizations of the relative importance of input features for a prediction. However, methods for creating maps often fail due to DNNs being sensitive to input noise, o…
National Digital Switching System Engineering and Technological Research Center（国家数字交换系统工程技术研究中心）；University of Electronic Science and Technology of China（电子科技大学）
In recent years, research on decoding brain activity based on functional magnetic resonance imaging (fMRI) has made remarkable achievements. However, constraint-free natural image reconstruction from brain activity is still a challenge. The existing methods simplified the problem by using semantic prior information or just reconstructing simple images such as letters and digitals. Without semantic prior information, we present a novel method to reconstruct nature images from fMRI signals of human visual cortex based on the computation model of convolutional neural network (CNN). Firstly, we ex…
University of Science and Technology of China（中国科学技术大学）
Decoding brain cognitive states from neuroimaging signals is an important topic in neuroscience. In recent years, deep neural networks (DNNs) have been recruited for multiple brain state decoding and achieved good performance. However, the open question of how to interpret the DNN black box remains unanswered. Capitalizing on advances in machine learning, we integrated attention modules into brain decoders to facilitate an in-depth interpretation of DNN channels. A 4D convolution operation was also included to extract temporo-spatial interaction within the fMRI signal. The experiments showed t…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Background: A universal unanswered question in neuroscience and machine learning is whether computers can decode the patterns of the human brain. Multi-Voxels Pattern Analysis (MVPA) is a critical tool for addressing this question. However, there are two challenges in the previous MVPA methods, which include decreasing sparsity and noise in the extracted features and increasing the performance of prediction. Methods: In overcoming mentioned challenges, this paper proposes Anatomical Pattern Analysis (APA) for decoding visual stimuli in the human brain. This framework develops a novel anatomica…
