Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
NeuBCI Lab, State Key Laboratory of Brain Cognition；Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China；School of Future Technology, University of Chinese Academy of Sciences, Beijing, China；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China；Zhongguancun Academy, Beijing, China；Peking University, Beijing, China
Yizhuo Lu, Changde Du, Qingyu Shi, Hang Chen, Jie Peng, Liuyun Jiang, Shuangchen Zhao, Huiguang He
Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen Memorial Hospital, Sun Yat-sen University（中山大学孙逸仙纪念医院）；Southern Medical University（南方医科大学）；Zhongshan Hospital, Fudan University（复旦大学中山医院）；Department of Neurosurgery, Prince of Wales Hospital（威尔士王室医院神经外科部）
OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens
Department of Ophthalmology, Byers Eye Institute, Stanford University（斯坦福大学眼科学系、比尔斯眼科研究所）；Stanford Bio-X, Stanford University（斯坦福大学生物交叉学科）；Wu Tsai Neurosciences Institute, Stanford University（斯坦福大学吴泰教授神经科学研究所）；Institute of Computer Science and Campus Institute Data Science, University Göttingen（哥廷根大学计算机科学研究所和校园数据科学研究所）
Tera-MIND: Tera-scale mouse brain simulation via spatial mRNA-guided diffusion
Department of Biomedical Engineering, University of Basel, Switzerland（巴塞尔大学生物医学工程系）；Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Switzerland（苏黎世大学病理学与分子病理学系，苏黎世大学医院）；Computer Vision Lab, ETH Zurich, Switzerland（苏黎世联邦理工学院计算机视觉实验室）；Integrated System Laboratory, ETH Zurich, Switzerland（苏黎世联邦理工学院集成系统实验室）；Institute of Medical Genetics and Pathology, University Hospital Basel, Switzerland（巴塞尔大学医学遗传学与病理学研究所，巴塞尔大学医院）
State University of New York at New Paltz（纽约州立大学新帕尔兹分校）；Northeastern University（东北大学）
Turing inspired a computer metaphor of the mind and brain that has been handy and has spawned decades of empirical investigation, but he did much more and offered behavioral and cognitive sciences another metaphor--that of the cascade. The time has come to confront Turing's cascading instability, which suggests a geometrical framework driven by power laws and can be studied using multifractal formalism and multiscale probability density function analysis. Here, we review a rapidly growing body of scientific investigations revealing signatures of cascade instability and their consequences for a…
Beijing University of Posts and Telecommunications（北京邮电大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）
Understanding how brain functions has been an intriguing topic for years. With the recent progress on collecting massive data and developing advanced technology, people have become interested in addressing the challenge of decoding brain wave data into meaningful mind states, with many machine learning models and algorithms being revisited and developed, especially the ones that handle time series data because of the nature of brain waves. However, many of these time series models, like HMM with hidden state in discrete space or State Space Model with hidden state in continuous space, only wor…
The brain is a powerful tool used to achieve amazing feats. There have been several significant advances in neuroscience and artificial brain research in the past two decades. This article is a review of such advances, ranging from the concepts of connectionism, to neural network architectures and high-dimensional representations. There have also been advances in biologically inspired cognitive architectures of which we will cite a few. We will be positioning relatively specific models in a much broader perspective, while comparing and contrasting their advantages and weaknesses. The projects …
Carnegie Mellon University（卡内基梅隆大学）；Jagiellonian University（雅盖隆大学）；Copernicus Center for Interdisciplinary Studies（哥白尼跨学科研究中心）
The interaction between syntax (formal language) and its semantics (meanings of language) is one which has been well studied in categorical logic. The results of this particular study are employed to understand how the brain is able to create meanings. To emphasize the toy character of the proposed model, we prefer to speak of the homunculus brain rather than the brain per se. The homunculus brain consists of neurons, each of which is modeled by a category, and axons between neurons, which are modeled by functors between the corresponding neuron-categories. Each neuron (category) has its own p…
Humboldt-University of Berlin（柏林洪堡大学）；University of Tartu（塔尔图大学）
Can deep learning (DL) guide our understanding of computations happening in biological brain? We will first briefly consider how DL has contributed to the research on visual object recognition. In the main part we will assess whether DL could also help us to clarify the computations underlying higher cognitive functions such as Theory of Mind. In addition, we will compare the objectives and learning signals of brains and machines, leading us to conclude that simply scaling up the current DL algorithms will most likely not lead to human level Theory of Mind.
IMT Lille Douai（IMT里尔杜埃）；CRIStAL UMR CNRS 9189（法国国家科学研究中心信息、自动化与信号处理实验室）；University of Mons（蒙斯大学）
Visual attention estimation is an active field of research at the crossroads of different disciplines: computer vision, artificial intelligence and medicine. One of the most common approaches to estimate a saliency map representing attention is based on the observed images. In this paper, we show that visual attention can be retrieved from EEG acquisition. The results are comparable to traditional predictions from observed images, which is of great interest. For this purpose, a set of signals has been recorded and different models have been developed to study the relationship between visual at…
Institute of Automation, CAS（中国科学院自动化研究所）；Beijing Institute of Technology（北京理工大学）
Reconstructing visual stimuli from measured functional magnetic resonance imaging (fMRI) has been a meaningful and challenging task. Previous studies have successfully achieved reconstructions with structures similar to the original images, such as the outlines and size of some natural images. However, these reconstructions lack explicit semantic information and are difficult to discern. In recent years, many studies have utilized multi-modal pre-trained models with stronger generative capabilities to reconstruct images that are semantically similar to the original ones. However, these images …
Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment
School of Robotics Xi’an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi'an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi’an Jiaotong-Liverpool University Suzhou, China；Data Science Research Center Duke Kunshan University Suzhou, China
MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data
Graduate School of Humanities and Sciences Ochanomizu University（大仓山大学人文科学研究生院）；Center for Information and Neural Networks Advanced ICT Research Institute National Institute of Information and Communications Technology（信息与神经网络中心先进信息通信研究机构信息通信技术研究所）
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Ecole Normale Supérieure, PSL University（巴黎高等师范学院，PSL大学）；University of Toronto（多伦多大学）；Hebrew University of Jerusalem（希伯来大学）；EleutherAI；Stability AI
There are many new forms of interfacing human users to machines. We persevere here electric mechanical form of interaction between human and machine. The emergence of brain-computer interface allows mind-to-movement systems. The story of the Pied Piper inspired us to devise some new heuristics for interfacing human motor system using brain waves by combining head helmet and LumbarMotionMonitor For the simulation we use java GridGain Brain responses of classified subjects during training indicates that Probe can be the best stimulus to rely on in distinguishing between knowledgeable and not kno…
An entirely novel synthesis combines the applied cognitive psychology of a task analytic approach with a neural cell assembly perspective that models both brain and mind function during task performance; similar cell assemblies could be implemented as an artificially intelligent neural network. A simplified cell assembly model is introduced and this leads to several new representational formats that, in combination, are demonstrated as suitable for analysing tasks. The advantages of using neural models are exposed and compared with previous research that has used symbolic artificial intelligen…
