基于Transformer的声学模型中的稀疏权重与边电路发现
Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models
浏览论文内容
中文总结 AI 辅助
本研究将电路发现框架DiscoGP扩展到语音编码器，发现极紧凑电路性能媲美完整模型，并推出内存高效变体，拓展了机制解释的应用范围。
中文摘要 AI 辅助
基于Transformer的基础模型功能强大但内部不透明，这促使人们采用机制解释方法来揭示黑箱，通过识别负责特定任务的小型计算子图来理解模型。DiscoGP是一个联合权重和边电路发现框架，最初是为文本解码器开发的。我们将DiscoGP扩展到语音编码器，并据我们所知，首次对现代语音基础模型进行了电路发现研究。在多个语音分类任务上，针对HuBERT和Wav2Vec 2.0模型，我们发现所发现的电路极其紧凑，但往往能达到甚至超过使用相同下游头部的完整预训练编码器的性能。通过消融实验，我们证明这些电路反映了预训练计算，而非随机结构或任务头部的伪影。我们还引入了一种内存高效的DiscoGP变体，将边电路发现过程中的GPU内存成本从四次方降低到三次方。总体而言，我们的研究将机制解释扩展到文本解码器之外，并表明电路级分析可以揭示语音编码器中的解释性结构和意想不到的功能行为。
英文摘要
Transformer-based foundation models are powerful but opaque, motivating Mechanistic Interpretation methods to uncover the black-box by identifying small computation subgraphs responsible for a task. DiscoGP is a joint weight-and-edge circuit discovery framework originally developed for text decoders. We extend DiscoGP to speech encoders and present, to our knowledge, the first circuit discovery study for modern speech foundation models. Across HuBERT and Wav2Vec 2.0 on several speech classification tasks, we find that the discovered circuits are extremely compact, yet often match or even exceed the performance of the full pretrained encoder with the same downstream head. Through ablations, we show that these circuits reflect pretrained computation rather than random structure or task-head artifacts. We also introduce a memory-efficient DiscoGP variant that reduces the GPU memory cost of edge-circuit discovery at runtime from quartic to cubic. Overall, our results broaden Mechanistic Interpretation beyond text decoders and show that circuit-level analysis can reveal both explanatory structure and unexpected functional behavior in speech encoders.
发表机构
- University of Toronto（多伦多大学）
机构由 AI 辅助整理，请以论文原文为准。
