Visual attention estimation is an active field of research at the crossroads of different disciplines: computer vision, artificial intelligence and medicine. One of the most common approaches to estimate a saliency map representing attention is based on the observed images. In this paper, we show that visual attention can be retrieved from EEG acquisition. The results are comparable to traditional predictions from observed images, which is of great interest. For this purpose, a set of signals has been recorded and different models have been developed to study the relationship between visual attention and brain activity. The results are encouraging and comparable with other approaches estimating attention with other modalities. The codes and dataset considered in this paper have been made available at \url{https://figshare.com/s/3e353bd1c621962888ad} to promote research in the field.
Tera-MIND: Tera-scale mouse brain simulation via spatial mRNA-guided diffusion
Department of Biomedical Engineering, University of Basel, Switzerland（巴塞尔大学生物医学工程系）；Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Switzerland（苏黎世大学病理学与分子病理学系，苏黎世大学医院）；Computer Vision Lab, ETH Zurich, Switzerland（苏黎世联邦理工学院计算机视觉实验室）；Integrated System Laboratory, ETH Zurich, Switzerland（苏黎世联邦理工学院集成系统实验室）；Institute of Medical Genetics and Pathology, University Hospital Basel, Switzerland（巴塞尔大学医学遗传学与病理学研究所，巴塞尔大学医院）
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University of Science and Technology（南京理工大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Agency for Science, Technology and Research（科技研究局）；New York University（纽约大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories. Afterwards, we train a Convolutional Neural Network (CNN)-based regressor to project images onto the learned manifold, thus effectively allowing machines to employ human brain-…
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Ecole Normale Supérieure, PSL University（巴黎高等师范学院，PSL大学）；University of Toronto（多伦多大学）；Hebrew University of Jerusalem（希伯来大学）；EleutherAI；Stability AI
Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applyin…
Institute of Automation, CAS（中国科学院自动化研究所）；Beijing Institute of Technology（北京理工大学）
Reconstructing visual stimuli from measured functional magnetic resonance imaging (fMRI) has been a meaningful and challenging task. Previous studies have successfully achieved reconstructions with structures similar to the original images, such as the outlines and size of some natural images. However, these reconstructions lack explicit semantic information and are difficult to discern. In recent years, many studies have utilized multi-modal pre-trained models with stronger generative capabilities to reconstruct images that are semantically similar to the original ones. However, these images …
Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment
School of Robotics Xi’an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi'an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi’an Jiaotong-Liverpool University Suzhou, China；Data Science Research Center Duke Kunshan University Suzhou, China
Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI
Obvious Research（奥布维研究机构）；GHU Paris Psychiatrie et Neurosciences（巴黎精神病学与神经科学大学医院集团）；Université Paris Cité（巴黎西岱大学）；Centre Imagerie du Nord（北方影像中心）；Raidium（雷迪姆公司）；FrontLab, Sorbonne Université（索邦大学前沿实验室）；Institut du Cerveau（大脑研究所）；Centre de NeuroImagerie de Recherche, Sorbonne Université（索邦大学神经影像研究中心）；Institut des Systèmes Intelligents et de la Robotique, Sorbonne Université（索邦大学智能系统与机器人研究所）
Drexel University（德雷塞尔大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Research has shown that neurons within the brain are selective to certain stimuli. For example, the fusiform face area (FFA) region is known by neuroscientists to selectively activate when people see faces over non-face objects. However, the mechanisms by which the primary visual system directs information to the correct higher levels of the brain are currently unknown. In our work, we mimic several high-level neural mechanisms of perception by creating a novel computational model that incorporates lateral and top down feedback in the form of hierarchical competition. Not only do we show that …
MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Stability AI；Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；University of Minnesota（明尼苏达大学）；The University of Sydney（悉尼大学）；University of Waterloo（滑铁卢大学）
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；Stanford University（斯坦福大学）
Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self…
Beihang University（北京航空航天大学）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Xiaohongshu Inc（小红书公司）；Zhongguancun Laboratory（中关村实验室）；Nanchang Institute of Technology（南昌理工学院）
Oslo Metropolitan University（奥斯陆城市大学）；Norwegian University of Science and Technology（挪威科技大学）
In the brain, the structure of a network of neurons defines how these neurons implement the computations that underlie the mind and the behavior of animals and humans. Provided that we can describe the network of neurons as a graph, we can employ methods from graph theory to investigate its structure or use cellular automata to mathematically assess its function. Although, software for the analysis of graphs and cellular automata are widely available. Graph extraction from the image of networks of brain cells remains difficult. Nervous tissue is heterogeneous, and differences in anatomy may re…
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Stanford University（斯坦福大学）；Stanford Stroke Center（斯坦福中风中心）；Institute for Biomedical Engineering（生物医学工程研究所）
We show that high quality, diverse and realistic-looking diffusion-weighted magnetic resonance images can be synthesized using deep generative models. Based on professional neuroradiologists' evaluations and diverse metrics with respect to quality and diversity of the generated synthetic brain images, we present two networks, the Introspective Variational Autoencoder and the Style-Based GAN, that qualify for data augmentation in the medical field, where information is saved in a dispatched and inhomogeneous way and access to it is in many aspects restricted.
In the last years we witness a dramatic growth of research focused on semantic image understanding. Indeed, without understanding image content successful accomplishment of any image-processing task is simply incredible. Up to the recent times, the ultimate need for such understanding has been met by the knowledge that a domain expert or a vision system supervisor have contributed to every image-processing application. The advent of the Internet has drastically changed this situation. Internet sources of visual information are diffused and dispersed over the whole Web, so the duty of informati…
In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the, typically huge, image collection on which the search is performed. We propose Text2Vis, a neural network that generates a visual representation, in the visual fea…
Cancer of the brain is deadly and requires careful surgical segmentation. The brain tumors were segmented using U-Net using a Convolutional Neural Network (CNN). When looking for overlaps of necrotic, edematous, growing, and healthy tissue, it might be hard to get relevant information from the images. The 2D U-Net network was improved and trained with the BraTS datasets to find these four areas. U-Net can set up many encoder and decoder routes that can be used to get information from images that can be used in different ways. To reduce computational time, we use image segmentation to exclude i…
