CORTIVA：用于EEG和MEG到图像检索的互补视觉教师的候选分数融合
CORTIVA: Candidate-Score Fusion of Complementary Visual Teachers for EEG- and MEG-to-Image Retrieval
浏览论文内容
中文总结 AI 辅助
CORTIVA是一种候选分数融合框架，通过保留互补视觉教师的异构证据，在EEG/MEG到图像检索任务中大幅提升了Top-1和Top-5准确率，为神经图像检索提供了更优方案。
中文摘要 AI 辅助
从无创脑活动中解码视觉体验是神经科学和脑机接口的核心问题。功能磁共振成像（fMRI）能提供精细的空间细节，但其缓慢的血液动力学和繁琐的采集过程限制了时间分辨率解码。脑电图（EEG）和脑磁图（MEG）具有毫秒级分辨率，因此图像检索极具吸引力：从一次神经响应和固定候选库中识别出被观看的图像。与预训练视觉表征的对比对齐可实现EEG和MEG的零样本检索，但大多数系统在排序前会将异构视觉监督合并为单个嵌入，这种早期合并对所有候选排序施加了单一相似性几何结构，并在最终排序中消除了编码器特定的分歧。我们提出CORTIVA，一种保留这种互补证据的候选分数融合框架。三条解码路径与异构视觉目标对齐，独立对相同索引的候选进行评分，仅在排序前合并它们经过温度缩放的分数向量。在200路THINGS-EEG2基准测试中，CORTIVA在10名参与者上达到73.5%的Top-1准确率和95.3%的Top-5准确率，比已报道的最强基线分别高出10.3和5.4个百分点。借助模态特定的神经编码器，相同的融合原理在THINGS-MEG上达到42.4%的Top-1准确率。匹配的路径移除重训练和四个权重控制实验表明，CORTIVA的增益源于整合互补路径分数，且在均匀加权时仍能保持，无需专门的加权规则。独立的DINOv2分析进一步复现了局部误差邻域和后验神经-视觉对应关系。这些结果表明，候选分数融合是神经图像检索中嵌入级合并的一种简单且可验证的替代方案。
英文摘要
Decoding visual experience from non-invasive brain activity is central to neuroscience and brain-computer interfaces. Functional magnetic resonance imaging (fMRI) offers fine spatial detail, but its slow hemodynamics and burdensome acquisition limit temporally resolved decoding. Electroencephalography (EEG) and magnetoencephalography (MEG) provide millisecond resolution, making image retrieval compelling: identify the viewed image from one neural response and a fixed candidate bank. Contrastive alignment to pretrained visual representations enables zero-shot retrieval from EEG and MEG, but most systems collapse heterogeneous visual supervision into a single embedding before ranking. This early consolidation imposes one similarity geometry on every candidate order and removes encoder-specific disagreements from the final ranking. We propose CORTIVA, a candidate-score fusion framework that preserves this complementary evidence. Three decoding routes are aligned to heterogeneous visual targets, score the same indexed candidates independently, and combine only their temperature-scaled score vectors before ranking. On the 200-way THINGS-EEG2 benchmark, CORTIVA reaches 73.5% Top-1 and 95.3% Top-5 across ten participants, exceeding the strongest reported baseline by 10.3 and 5.4 percentage points. With a modality-specific neural encoder, the same fusion principle reaches 42.4% Top-1 on THINGS-MEG. Matched route-removal retraining and four weight controls demonstrate that CORTIVA's gain arises from integrating complementary route scores and persists with uniform weighting, without requiring a specialized weighting rule. Independent DINOv2 analyses further reproduce the local error neighborhoods and posterior neural-visual correspondence. These results establish candidate-score fusion as a simple and testable alternative to embedding-level consolidation for neural image retrieval.
发表机构
- The Hong Kong University of Science and Technology（香港科技大学）
机构由 AI 辅助整理，请以论文原文为准。
