面向音频频谱变换器中更好录音设备泛化的频率维度归一化研究
On Frequency-Wise Normalizations for Better Recording Device Generalization in Audio Spectrogram Transformers
AI总结：
本研究针对声学场景分类中录音设备不匹配问题，提出对音频频谱变换器输入进行频率居中操作，以抑制设备特征，在未见设备上平均提升性能达18.2个百分点。
摘要：
训练数据与测试数据之间的条件差异仍然是机器学习面临的主要挑战。我们在声学场景分类（ASC）任务中研究了这一问题，并考虑了录音设备不匹配的情况。以往的研究成功地在卷积神经网络中对输入和隐藏层激活进行频率维度归一化，以减少录音设备差异带来的影响。本研究的主要目标是将频率维度归一化应用于音频频谱变换器（ASTs），后者近年来已成为ASC中的主导模型架构。为此，我们首先调查了录音设备特征是如何编码在AST的隐藏层激活中的。我们发现，录音设备信息最初编码在频率维度上；然而，在第一个自注意力块之后，这些信息大部分被转换到令牌维度。基于这一观察，我们推测在输入频谱图中抑制录音设备特征是最有效的方法。我们提出了一种针对频谱图的频率居中操作，该操作在未见过的录音设备上平均将ASC性能提升了高达18.2个百分点。
英文摘要：
Varying conditions between the data seen at training and at application time remain a major challenge for machine learning. We study this problem in the context of Acoustic Scene Classification (ASC) with mismatching recording devices. Previous works successfully employed frequency-wise normalization of inputs and hidden layer activations in convolutional neural networks to reduce the recording device discrepancy. The main objective of this work was to adopt frequency-wise normalization for Audio Spectrogram Transformers (ASTs), which have recently become the dominant model architecture in ASC. To this end, we first investigate how recording device characteristics are encoded in the hidden layer activations of ASTs. We find that recording device information is initially encoded in the frequency dimension; however, after the first self-attention block, it is largely transformed into the token dimension. Based on this observation, we conjecture that suppressing recording device characteristics in the input spectrogram is the most effective. We propose a frequency-centering operation for spectrograms that improves the ASC performance on unseen recording devices on average by up to 18.2 percentage points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
