arXiv 2605.082142026-05-06cs.SDcs.AIeess.AS

Bangla-WhisperDiar: 通过微调Whisper和PyAnnote实现孟加拉语长篇语音识别与说话人辨识

Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

Department of ECE, North South University（北南大学电子与计算机工程系）

Mohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty, Aritra Islam Saswato, Ahmed Faizul Haque Dhrubo, Mohammad Ashrafuzzaman Khan

备注：: 3 figures and 5 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过微调Whisper和PyAnnote模型，解决孟加拉语长篇语音识别和说话人辨识的挑战，实现了WER 0.2441和DER 0.2392的性能提升。

摘要：

自动语音识别（ASR）和说话人辨识在孟加拉语中仍然具有挑战性，由于长篇录音、多样的声学条件和显著的说话人差异。本文通过开发稳健的长篇ASR和说话人辨识系统，解决了这两个核心任务。对于ASR（问题1），我们微调了tugstugi bengaliai区域ASR Whisper medium模型，使用约15,000个分块和对齐的孟加拉语音频片段的定制数据集，采用全权重训练并使用大量数据增强，包括噪声注入、回声模拟、回声、剪切失真和音高/时间扰动。对于说话人辨识（问题2），我们使用PyTorch Lightning微调pyannote/segmentation-3.0模型，将微调的分割骨干网络替换到pyannote/speaker-diarization-community-1流程中，同时保留预训练的说话人嵌入和聚类组件。我们的ASR系统在测试集上实现了2.441%的词错误率（WER），而我们的辨识系统实现了2.392%的辨识错误率（DER），均显著优于相应的预训练基线。我们描述了完整的流程，包括数据预处理、文本标准化、音频增强、训练策略、推理优化和后处理。

英文摘要：

Automatic Speech Recognition (ASR) and speaker diarization in Bangla remain challenging due to long form recordings, diverse acoustic conditions, and significant speaker variability. This work addresses these two core tasks in Bangla spoken language understanding by developing robust systems for long form ASR and speaker diarization. For ASR (Problem 1), we fine tune the tugstugi bengaliai regional asr whisper medium model on a custom-curated dataset of approximately 15,000 chunked and aligned Bangla audio segments, employing full weight training with extensive data augmentation including noise injection, reverb simulation, echo, clipping distortion, and pitch/time perturbation. For speaker diarization (Problem 2), we fine-tune the pyannote/segmentation-3.0 model using PyTorch Lightning on the competition annotated diarization dataset, swapping the fine-tuned segmentation backbone into the pyannote/speaker-diarization-community-1 pipeline while retaining the pretrained speaker embedding and clustering components. Our ASR system achieves a Word Error Rate (WER) of 0.2441, while our diarization system achieves a Diarization Error Rate (DER) of 0.2392, both evaluated on the test set, demonstrating notable improvements over the respective pretrained baselines. We describe our complete pipeline, including data preprocessing, text normalization, audio augmentation, training strategies, inference optimization, and post-processing for both tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。