基于多视图伪标签的语音半监督认知状态分类
Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labeling
AI总结：
针对认知状态分类中标注数据缺乏的问题，提出一种结合声学和语言特征的多视图伪标签半监督学习框架，仅用30%标注数据即达到与全监督学习相当的竞争力并显著优于基线。
摘要：
缺乏标注数据是语音分类任务中常见的挑战，尤其是在需要大量主观评估的任务中，如认知状态分类。在本研究中，我们提出了一种半监督学习（SSL）框架，引入了一种新颖的多视图伪标签方法，该方法同时利用声学和语言特征来选择最可信的数据用于训练分类模型。在声学方面，使用由多个音频编码器生成的嵌入计算出的Frechet音频距离，将未标注数据与已标注数据进行比较。在语言方面，提示大型语言模型修正自动语音识别转录文本，并根据我们提出的特定任务知识预测标签。当两个来源的伪标签一致时，数据被识别为高置信度，而不匹配则被视为低置信度数据。随后训练一个双模态分类器，迭代标注低置信度数据，直到满足预定义标准。我们在情感识别和痴呆症检测任务上评估了我们的SSL框架。实验结果表明，与使用100%标注数据的全监督学习相比，我们的方法仅使用30%的标注数据即可达到具有竞争力的性能，并且显著优于两个选定的基线方法。
英文摘要：
The lack of labeled data is a common challenge in speech classification tasks, particularly those requiring extensive subjective assessment, such as cognitive state classification. In this work, we propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method that leverages both acoustic and linguistic characteristics to select the most confident data for training the classification model. Acoustically, unlabeled data are compared to labeled data using the Frechet audio distance, calculated from embeddings generated by multiple audio encoders. Linguistically, large language models are prompted to revise automatic speech recognition transcriptions and predict labels based on our proposed task-specific knowledge. High-confidence data are identified when pseudo-labels from both sources align, while mismatches are treated as low-confidence data. A bimodal classifier is then trained to iteratively label the low-confidence data until a predefined criterion is met. We evaluate our SSL framework on emotion recognition and dementia detection tasks. Experimental results demonstrate that our method achieves competitive performance compared to fully supervised learning using only 30% of the labeled data and significantly outperforms two selected baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
