The study of decoding visual neural information faces challenges in generalizing single-subject decoding models to multiple subjects, due to individual differences. Moreover, the limited availability of data from a single subject has a constraining impact on model performance. Although prior multi-subject decoding methods have made significant progress, they still suffer from several limitations, including difficulty in extracting global neural response features, linear scaling of model parameters with the number of subjects, and inadequate characterization of the relationship between neural responses of different subjects to various stimuli. To overcome these limitations, we propose a CLIP-guided Multi-sUbject visual neural information SEmantic Decoding (CLIP-MUSED) method. Our method consists of a Transformer-based feature extractor to effectively model global neural representations. It also incorporates learnable subject-specific tokens that facilitates the aggregation of multi-subject data without a linear increase of parameters. Additionally, we employ representational similarity analysis (RSA) to guide token representation learning based on the topological relationship of visual stimuli in the representation space of CLIP, enabling full characterization of the relationship between neural responses of different subjects under different stimuli. Finally, token representations are used for multi-subject semantic decoding. Our proposed method outperforms single-subject decoding methods and achieves state-of-the-art performance among the existing multi-subject methods on two fMRI datasets. Visualization results provide insights into the effectiveness of our proposed method. Code is available at https://github.com/CLIP-MUSED/CLIP-MUSED.
Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associations
Institute for Interdisciplinary Studies, University of Amsterdam（阿姆斯特丹大学跨学科研究所）；Department of Information and Computing Sciences, Utrecht University（乌得勒支大学信息与计算科学系）
VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI
Department of Electrical and Computer Engineering, New York University Tandon School of Engineering（电气工程系，纽约大学塔能工程学院）；Department of Computer Science and Engineering, New York University Tandon School of Engineering（计算机科学与工程系，纽约大学塔能工程学院）
Brain3D: EEG-to-3D Decoding of Visual Representations via Multimodal Reasoning
Università Politecnica delle Marche（马尔凯理工大学）；University of Macerata（马切拉塔大学）；Horizon Intelligence Labs（地平线智能实验室）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）
Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与脑启发智能技术国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models
Jinan University, Guangzhou, China；Department of Rehabilitation Medicine, The First Affiliated Hospital of Jinan University, Guangzhou, China；Western University, Ontario, Canada；Guangdong University of Technology, Guangzhou, China；Tsinghua University, Beijing, China
NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenes
Department of Electronic and Electrical Engineering, Southern University of Science and Technology（电子与电气工程系，南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）
Entropy-Aware Structural Alignment for Zero-Shot Handwritten Chinese Character Recognition
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Shenzhen Key Laboratory of Embedded System Design（深圳嵌入式系统设计重点实验室）；Undergraduate School of Artificial Intelligence, Shenzhen Polytechnic University（深圳职业技术学院人工智能本科生学院）；Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong Macao Greater Bay Area（粤港澳大湾区应用人工智能研究院）
Using Multimodal Deep Neural Networks to Disentangle Language from Visual Aesthetics
Harvard University（哈佛大学）；Harvard School of Public Health（哈佛公共卫生学院）；Massachusetts General Hospital（麻省总医院）；Massachusetts Institute of Technology（麻省理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
