arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.06392cs.SD
收藏

InsideSSL:从模型中心视角理解自监督语音表示

InsideSSL: Understanding Self-Supervised Speech Representations using a Model-Centric Perspective

Inria, Université Grenoble Alpes CNRS, LJK(法国国家信息与自动化研究所、格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、法国国家科学研究中心计算机图形学实验室)

Samir Sadok, Xavier Alameda-Pineda

AI总结:

研究旨在理解自监督语音表示模型内部动态,提出InsideSSL框架,从压缩、几何等视角分析,引入GCM评估,通过线性探测连接到下游任务,展示层拓扑对语音相关编码的作用。

摘要:

自监督学习(SSL)模型,如Wav2Vec2、HuBERT和WavLM,已成为众多语音和音频任务的基础。尽管取得了成功,但理解其内部逐层动态仍是一项持续的挑战。为此,我们提出了一个名为InsideSSL的以模型为中心的两部分框架。首先,我们从三个内在的逐层视角进行与任务无关的分析:压缩(熵)、几何(曲率)和对扰动的鲁棒性。我们表明,不同的训练目标会导致不同的声学压缩和流形展开状态。其次,我们引入跨层生成兼容性矩阵(GCM)来评估功能可转移性,揭示稳定的语音核心、身份波动性和深层语义修剪。除了这些评估之外,线性探测将以模型为中心的视角与下游任务联系起来,展示了层拓扑如何决定音素、音高和说话者编码。

英文摘要:

Self-supervised learning (SSL) models, such as Wav2Vec2, HuBERT, and WavLM, have become foundational across a wide range of speech and audio tasks. Despite their success, understanding their internal layer-wise dynamics remains an ongoing challenge. To address this, we propose a two-part model-centric framework called InsideSSL. First, we establish a task-agnostic analysis from three intrinsic per-layer perspectives: compression (entropy), geometry (curvature), and robustness to perturbations. We show that varying training objectives induce distinct regimes of acoustic compression and manifold unfolding. Second, we introduce the cross-layer Generative Compatibility Matrix (GCM) to evaluate functional transferability, exposing stable phonetic cores, identity volatility, and deep-layer semantic pruning. In addition to these evaluations, linear probing connects the model-centric perspective to downstream tasks, demonstrating how layer topology dictates phoneme, pitch, and speaker encoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑