arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01247cs.LGcs.AI

跨语音模式的脑-语音模型机制可解释性

Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes

Department of Computer Science, University of Maryland, College Park, United States(计算机科学系,马里兰大学,College Park,美国);Department of Electrical and Computer Engineering, University of Maryland, College Park, United States(电气与计算机工程系,马里兰大学,College Park,美国)

Maryam Maghsoudi, Ayushi Mishra

AI总结:

本研究通过机制可解释性方法揭示了脑-语音解码模型中语音模态信息的组织与利用机制,发现语音模态位于共享的连续因果流形上,跨模态转移由紧凑的层特定子空间介导。

摘要:

脑-语音解码模型在发声、模仿和想象语音中表现出稳健的性能;然而,这些模型如何在不同语音模态之间捕捉和传输信息的基本机制仍较少被探索。在本工作中,我们使用机制可解释性来因果地研究神经语音解码器的内部表示。我们跨语音模态对内部激活进行激活补丁操作,并使用三模态插值来检验语音表示是否以离散或连续的方式变化。我们使用粗到细的因果追踪和因果刮擦来发现局部因果结构,使我们能够找到足够用于跨模态转移的内部子空间。为了确定这些效果在层内的分布程度,我们进行了神经元级别的激活补丁操作。我们发现,小但不分散的神经元子集,而不是孤立单元,影响跨模态转移。我们的结果表明,语音模态位于共享的连续因果流形上,跨模态转移由紧凑的、层特定的子空间而非扩散活动所中介。共同,我们的发现为脑-语音解码模型如何组织和使用语音模态信息提供了因果解释,揭示了不同语音模态之间的层次性和方向依赖性表示结构。

英文摘要:

Brain-to-speech decoding models demonstrate robust performance in vocalized, mimed, and imagined speech; yet, the fundamental mechanisms via which these models capture and transmit information across different speech modalities are less explored. In this work, we use mechanistic interpretability to causally investigate the internal representations of a neural speech decoder. We perform cross-mode activation patching of internal activations across speech modes, and use tri-modal interpolation to examine whether speech representations vary discretely or continuously. We use coarse-to-fine causal tracing and causal scrubbing to find localized causal structure, allowing us to find internal subspaces that are sufficient for cross-mode transfer. In order to determine how finely distributed these effects are within layers, we perform neuron-level activation patching. We discover that small but not distributed subsets of neurons, rather than isolated units, affect the cross-mode transfer. Our results show that speech modes lie on a shared continuous causal manifold, and cross-mode transfer is mediated by compact, layer-specific subspaces rather than diffuse activity. Together, our findings give a causal explanation for how speech modality information is organized and used in brain-to-speech decoding models, revealing hierarchical and direction-dependent representational structure across speech modes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑