arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.08068cs.CV

Brain3D: 通过多模态推理实现EEG到3D视觉表示的解码

Brain3D: EEG-to-3D Decoding of Visual Representations via Multimodal Reasoning

Università Politecnica delle Marche(马尔凯理工大学);University of Macerata(马切拉塔大学);Horizon Intelligence Labs(地平线智能实验室);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院)

Emanuele Balloni, Emanuele Frontoni, Chiara Matti, Marina Paolanti, Roberto Pierdicca, Emiliano Santarnecchi

AI总结:

本文提出Brain3D,通过多模态推理实现EEG到3D解码,分阶段生成3D网格,提升神经解码的几何理解与应用性。

摘要:

从脑电图(EEG)解码视觉信息近年来取得了显著进展,主要集中在从脑活动重建二维(2D)图像。然而,三维(3D)表示的重建仍鲜有研究,限制了几何理解并降低了神经解码在不同场景中的适用性。为此,我们提出Brain3D,一种基于EEG到图像解码的多模态架构,通过几何感知的生成推理逐步将神经表示转换到3D领域。我们的流程首先从EEG信号生成视觉基础图像,然后利用多模态大语言模型提取结构化的3D感知描述,这些描述指导基于扩散的生成阶段,最终通过单图像到3D模型转换为连贯的3D网格。通过将问题分解为结构化阶段,所提方法避免了直接的EEG到3D映射,实现了可扩展的脑驱动3D生成。我们对重建的3D输出与原始视觉刺激进行综合评估,评估语义对齐和几何保真度。实验结果表明,所提架构表现优异,达到最高85.4%的10-way Top-1 EEG解码准确率和0.648 CLIPScore,支持了多模态EEG驱动3D重建的可行性。

英文摘要:

Decoding visual information from electroencephalography (EEG) has recently achieved promising results, primarily focusing on reconstructing two-dimensional (2D) images from brain activity. However, the reconstruction of three-dimensional (3D) representations remains largely unexplored. This limits the geometric understanding and reduces the applicability of neural decoding in different contexts. To address this gap, we propose Brain3D, a multimodal architecture for EEG-to-3D reconstruction based on EEG-to-image decoding. It progressively transforms neural representations into the 3D domain using geometry-aware generative reasoning. Our pipeline first produces visually grounded images from EEG signals, then employs a multimodal large language model to extract structured 3D-aware descriptions, which guide a diffusion-based generation stage whose outputs are finally converted into coherent 3D meshes via a single-image-to-3D model. By decomposing the problem into structured stages, the proposed approach avoids direct EEG-to-3D mappings and enables scalable brain-driven 3D generation. We conduct a comprehensive evaluation comparing the reconstructed 3D outputs against the original visual stimuli, assessing both semantic alignment and geometric fidelity. Experimental results demonstrate strong performance of the proposed architecture, achieving up to 85.4% 10-way Top-1 EEG decoding accuracy and 0.648 CLIPScore, supporting the feasibility of multimodal EEG-driven 3D reconstruction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑