HDND：用于从非侵入性脑记录中进行多语言单词/字符检索的层次化动态神经解码
HDND: Hierarchical Dynamic Neural Decoding for Multilingual Word/Character Retrieval from Non-Invasive Brain Recordings
- The Hong Kong Polytechnic University（香港理工大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出HDND框架，通过层次化动态解码和残差精炼，从非侵入性脑记录中提升多语言单词检索性能，在九种条件下多数优于基线。
AI中文摘要：
虽然深度学习已能从颅内脑记录中实现语言解码，但将这一能力扩展到非侵入性记录仍然是一个未解决的挑战。从非侵入性脑记录中解码单个单词尤其困难，因为单词级别的神经证据较弱、在时间上分布广泛，并与声学、词汇和语义结构交织在一起。现有的检索流程往往将这些因素压缩为单一表示，可能丢弃中间时间尺度上可用的信息。在此，我们引入了层次化动态神经解码（HDND），这是一种层次化动态解码框架，将单词解码视为结构化精炼而非扁平标签检索。HDND结合了中间神经表示、上下文语义预测，并在选定的阅读条件下，结合了辅助的字符形式目标。我们在涵盖英语、荷兰语、普通话和粤语的听、读和朗读条件的七个脑电图（EEG）和脑磁图（MEG）数据集上评估了HDND。在九种条件的单词检索基准中，所提出的HDND在每种条件下都比匹配的上下文单词解码基线产生更高的参与者平均平衡Top-10点估计，并在九种条件中的八种中实现了所有比较方法中的最高平均值。在相同的九个匹配条件下，HDND在每种设置中还产生了更高的令牌微观和池化单词宏观Top-10点估计。句子检索在九种条件中的八种中偏向HDND，而听觉语音片段检索在六种听力条件下结果不一。这些结果表明，层次化残差精炼可以改善来自异质非侵入性脑记录的多语言单词检索。
英文摘要：
While deep learning has enabled language decoding from intracranial brain recordings, extending this capability to non-invasive recordings remains an unresolved challenge. Decoding individual words from non-invasive brain recordings is particularly difficult, as word-level neural evidence is weak, temporally distributed, and entangled with acoustic, lexical, and semantic structure. Existing retrieval pipelines often collapse these factors into a single representation, potentially discarding information available at intermediate temporal scales. Here, we introduce Hierarchical Dynamic Neural Decoding (HDND), a hierarchical dynamic decoding framework that treats word decoding as structured refinement rather than flat label retrieval. HDND combines intermediate neural representations, contextual semantic predictions, and, for selected reading conditions, an auxiliary character-form objective. We evaluate HDND across seven electroencephalography (EEG) and magnetoencephalography (MEG) datasets spanning English, Dutch, Mandarin, and Cantonese listening, reading, and reading-aloud conditions. Across the nine-condition word-retrieval benchmark, the proposed HDND yields a higher participant-averaged balanced Top-10 point estimate than the matched contextual word-decoding baseline in every condition and achieves the highest mean among all compared methods in eight of nine conditions. Across the same nine matched conditions, HDND also yields higher token-micro and pooled word-macro Top-10 point estimates in every setting. Sentence retrieval favors HDND in eight of nine conditions, while auditory speech-segment retrieval is mixed across the six listening conditions. These results show that hierarchical residual refinement can improve multilingual word retrieval from heterogeneous non-invasive brain recordings.
