机制可解释性揭示脑到语音解码器中的共享因果子空间

中文摘要 AI 辅助

从大脑活动中解码隐蔽言语（如默读或想象言语）比解码发声言语更为困难。跨模态迁移，即一种言语形式的信息帮助解码另一种形式，是一种有前景的补救措施；然而，解码器如何在内部表示和处理来自不同言语形式的大脑活动仍不清楚。在这项工作中，我们提出疑问：解码器的哪些内部神经元携带跨模态信息，这些神经元在不同言语形式之间是否共享？为回答这些问题，我们利用机制可解释性，使用同一句子的发声、默读和想象输入对进行激活修补。我们将解码器在一个条件下对某句子的内部活动插入到其对另一条件下同一句子的处理中，并测量解码精度的变化。我们发现，没有单个神经元驱动这种益处；相反，它来自小群神经元，其中发声言语是最有用的来源。这些群体在解码器的早期阶段主要是条件特异性的，但在后期阶段重叠。这些发现指向通过鼓励共享后期表征的训练目标（主要从发声数据学习）来实现更数据高效的隐蔽言语解码器。

英文摘要

Decoding covert speech, such as mimed or imagined, from brain activity is harder than decoding vocalized speech. Cross-modal transfer, where information from one speech form helps decode another, is a promising remedy; yet how a decoder internally represents and processes brain activity from different speech forms remains unclear. In this work, we ask: which internal neurons of a decoder carry cross-modal information, and are these neurons shared across different speech forms? To answer these questions, we leverage mechanistic interpretability, using recordings of the same sentences in vocalized, mimed, and imagined input pairs for activation patching. We insert the decoder's internal activity for a sentence in one condition into its processing of the same sentence in another and measure the change in decoding accuracy. We find that no single neuron drives this benefit; instead, it arises from small groups of neurons, with vocalized speech as the most useful source. These groups are largely condition-specific in the early stage of the decoder but overlap in the later stage. These findings point toward more data-efficient covert speech decoders through training objectives that encourage shared later-stage representations learned mainly from vocalized data.

发表机构

• University of Maryland, College Park（马里兰大学帕克分校）

机构由 AI 辅助整理，请以论文原文为准。