InsideSSL:从模型中心视角理解自监督语音表示
InsideSSL: Understanding Self-Supervised Speech Representations using a Model-Centric Perspective
AI总结:
研究旨在理解自监督语音表示模型内部动态,提出InsideSSL框架,从压缩、几何等视角分析,引入GCM评估,通过线性探测连接到下游任务,展示层拓扑对语音相关编码的作用。
摘要:
自监督学习(SSL)模型,如Wav2Vec2、HuBERT和WavLM,已成为众多语音和音频任务的基础。尽管取得了成功,但理解其内部逐层动态仍是一项持续的挑战。为此,我们提出了一个名为InsideSSL的以模型为中心的两部分框架。首先,我们从三个内在的逐层视角进行与任务无关的分析:压缩(熵)、几何(曲率)和对扰动的鲁棒性。我们表明,不同的训练目标会导致不同的声学压缩和流形展开状态。其次,我们引入跨层生成兼容性矩阵(GCM)来评估功能可转移性,揭示稳定的语音核心、身份波动性和深层语义修剪。除了这些评估之外,线性探测将以模型为中心的视角与下游任务联系起来,展示了层拓扑如何决定音素、音高和说话者编码。
英文摘要:
Self-supervised learning (SSL) models, such as Wav2Vec2, HuBERT, and WavLM, have become foundational across a wide range of speech and audio tasks. Despite their success, understanding their internal layer-wise dynamics remains an ongoing challenge. To address this, we propose a two-part model-centric framework called InsideSSL. First, we establish a task-agnostic analysis from three intrinsic per-layer perspectives: compression (entropy), geometry (curvature), and robustness to perturbations. We show that varying training objectives induce distinct regimes of acoustic compression and manifold unfolding. Second, we introduce the cross-layer Generative Compatibility Matrix (GCM) to evaluate functional transferability, exposing stable phonetic cores, identity volatility, and deep-layer semantic pruning. In addition to these evaluations, linear probing connects the model-centric perspective to downstream tasks, demonstrating how layer topology dictates phoneme, pitch, and speaker encoding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。