arXiv 2509.007872026-02-13cs.CV

基于CLIP引导的多模态扩散模型的图像到脑信号生成用于视觉假体

Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models

Jinan University, Guangzhou, China；Department of Rehabilitation Medicine, The First Affiliated Hospital of Jinan University, Guangzhou, China；Western University, Ontario, Canada；Guangdong University of Technology, Guangzhou, China；Tsinghua University, Beijing, China

Ganxi Xu, Zhao-Rong Lai, Yuting Tang, Yonghao Song, Guoxu Zhou, Boyu wang, Jian Zhu, Jinyi Long

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于CLIP引导的多模态扩散模型，实现图像到脑信号的转换，通过交叉注意力机制和空间-时间位置编码生成生物合理的脑信号。

摘要：

视觉假体在恢复盲人视力方面具有巨大潜力。尽管研究人员已成功利用M/EEG信号在视觉假体的脑解码阶段唤起视觉感知，但在脑编码阶段将图像转换为M/EEG信号的互补过程仍 largely 未被探索，阻碍了完整功能流程的形成。在本工作中，我们提出了一种新颖的图像到脑信号框架，通过利用增强交叉注意力机制的扩散变压器架构来从图像生成M/EEG信号。具体而言，我们采用基于去噪扩散隐式模型（DDIM）的扩散变压器（DiT）架构来实现脑信号生成。为了实现图像到脑信号转换的目标，我们使用交叉注意力机制对脑信号嵌入与CLIP图像嵌入进行对齐。此外，我们利用大型语言模型（LLMs）生成图像描述，并将生成的CLIP文本嵌入与CLIP图像嵌入连接起来，形成统一的嵌入用于交叉注意力对齐，使我们的模型能够捕捉核心语义信息。此外，为了捕捉核心语义信息，我们使用大型语言模型（LLMs）为图像生成描述性且语义准确的描述。进一步地，我们引入了可学习的空间-时间位置编码，将脑区嵌入与时间嵌入结合，以捕捉脑信号的空间和时间特性。我们评估了该框架在两个多模态基准数据集（THINGS-EEG2和THINGS-MEG）上的表现，并证明其生成生物合理的脑信号。

英文摘要：

Visual prostheses hold great promise for restoring vision in blind individuals. While researchers have successfully utilized M/EEG signals to evoke visual perceptions during the brain decoding stage of visual prostheses, the complementary process of converting images into M/EEG signals in the brain encoding stage remains largely unexplored, hindering the formation of a complete functional pipeline. In this work, we present a novel image-to-brain signal framework that generates M/EEG from images by leveraging the diffusion transformer architecture enhanced with cross-attention mechanisms. Specifically, we employ a diffusion transformer (DiT) architecture based on denoising diffusion implicit models (DDIM) to achieve brain signal generation. To realize the goal of image-to-brain signal conversion, we use cross-attention mechanisms to align brain signal embeddings with CLIP image embeddings. Moreover, we leverage large language models (LLMs) to generate image captions, and concatenate the resulting CLIP text embeddings with CLIP image embeddings to form unified embeddings for cross-attention alignment, enabling our model to capture core semantic information. Moreover, to capture core semantic information, we use large language models (LLMs) to generate descriptive and semantically accurate captions for images. Furthermore, we introduce a learnable spatio-temporal position encoding that combines brain region embeddings with temporal embeddings to capture both spatial and temporal characteristics of brain signals. We evaluate the framework on two multimodal benchmark datasets (THINGS-EEG2 and THINGS-MEG) and demonstrate that it generates biologically plausible brain signals.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。