arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.11440cs.CL
收藏

LAHAJA:用于评估印地语ASR系统的鲁棒多口音基准

LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems

AI4Bharat;Indian Institute of Technology Madras(印度马德拉斯理工学院)

Tahir Javed, Janki Nawale, Sakshi Joshi, Eldho George, Kaushal Bhogale, Deovrat Mehendale, Mitesh M. Khapra

AI总结:

针对印地语多口音ASR评估,构建LAHAJA基准(12.5小时音频,132位说话者),发现现有模型性能差,而基于多语言且说话者多样的训练模型显著更优,且东北和南部口音性能下降明显。

摘要:

印地语是印度使用最广泛的语言之一,由于使用者来自不同的语言背景,其口音呈现出多样性。为了对多口音下的印地语ASR系统进行鲁棒评估,我们创建了基准LAHAJA,其中包含多种主题和用例的朗读及即兴语音,总计12.5小时的印地语音频,来自印度83个地区的132位说话者。我们评估了现有的开源和商业模型在LAHAJA上的表现,发现其性能较差。随后,我们使用不同数据集训练模型,发现基于多语言数据并具有良好说话者多样性的模型显著优于现有模型。我们还进行了细粒度分析,结果显示来自印度东北部和南部的说话者性能下降,尤其是在包含大量命名实体和专业术语的内容上。

英文摘要:

Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑