发表机构
University of Kentucky; Wheaton College（肯塔基大学; 惠顿学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出用CLIP对比损失替代MSE训练Bi-LSTM解码器，从fNIRS信号重建单词语义，实验表明CLIP目标更稳定，为神经语义解码提供新方向。
AI 中文摘要
语义重建将神经活动映射到词嵌入空间，恢复感知单词的含义，而不是从固定词汇表中进行选择。功能性近红外光谱（fNIRS）携带适用于这种映射的语义信息。然而，大多数fNIRS解码器使用平方误差目标进行训练，该目标独立拟合每个单词，忽略了嵌入空间的几何结构。为解决这一局限，我们评估了一种基于对比语言-图像预训练（CLIP）损失的对比损失，作为均方误差（MSE）的替代方案，用于从fNIRS信号重建感知单词。我们通过训练一个双向长短期记忆（Bi-LSTM）解码器将fNIRS信号映射到词嵌入，来比较这两种目标。我们使用GloVe-50和T5词嵌入作为目标，在三个共享范式（将每个单词图像与其口语名称配对）下记录的fNIRS数据集上进行实验。性能通过成对匹配分数和开放词汇top-k检索来衡量。使用CLIP训练的Bi-LSTM是跨实验最一致的解码器。T5产生更高的匹配分数，而每个显著的检索结果都使用GloVe-50。这些结果支持使用对比目标作为fNIRS语义解码的一个有前景的方向，并激励在更大数据集上进行验证。
英文摘要
Semantic reconstruction maps neural activity to a word-embedding space, recovering the meaning of a perceived word instead of selecting it from a fixed vocabulary. Functional near-infrared spectroscopy (fNIRS) carries semantic information suitable for this mapping. However, most fNIRS decoders are trained with a squared-error objective that fits each word independently and ignores the geometry of the embedding space. To address this limitation, we evaluate a contrastive loss based on the contrastive-language-image-pretraining (CLIP) loss, as an alternative to mean-squared-error (MSE) for reconstructing perceived words from fNIRS. We compare the two objectives by training a bidirectional long short-term memory (Bi-LSTM) decoder to map fNIRS signals to word embeddings. We use GloVe-50 and T5 word embeddings as targets, across three fNIRS datasets recorded under a shared paradigm pairing each word image with its spoken name. Performance is measured with a pairwise matching score and open-vocabulary top-$k$ retrieval. The Bi-LSTM trained with CLIP is the most consistent decoder across experiments. T5 produces higher matching scores, whereas every significant retrieval result uses GloVe-50. These results support the use of contrastive objectives as a promising direction for fNIRS semantic decoding and motivate validation on larger datasets.
