arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14062cs.CLcs.SD
收藏

从匮乏到规模:对帕什托语Common Voice数据集的释放级分析

From Scarcity to Scale: A Release-Level Analysis of the Pashto Common Voice Dataset

O.P. Jindal Global University(O.P.贾丁格洛布大学);Technical University of Munich(慕尼黑技术大学)

Jandad Jahani, Mursal Dawodi, Jawid Ahmad Baktash

AI总结:

本文分析了帕什托语Common Voice数据集的规模增长及验证情况,揭示了参与不均和元数据不足的问题,提出提升数据集质量的改进方向。

摘要:

大规模、公开授权的语音数据集对于构建自动语音识别(ASR)系统至关重要,然而许多广泛使用的语言在公共资源中仍然代表性不足。帕什托语,由超过6000万人使用,历史上缺乏适合现代ASR开发的大规模公开授权语音数据。本文对Mozilla Common Voice语料库中的帕什托语部分进行了释放级分析,重点关注版本24.0(2025年12月)以及主要版本中的趋势。我们记录了从2023年中期的1.49小时录制时间增长到2025年的2768.7小时总时间,其中包括975.89小时可用于监督ASR训练的验证时间。除了规模之外,我们还分析了验证吞吐量、贡献者参与不平等、人口统计数据的完整性以及验证子集中的句子级集中情况。我们发现参与度极不均衡(基尼系数=0.941),年龄代表性强烈偏向年轻人,且41.97%的片段缺乏自我报告的性别标签,限制了基于元数据的子组审计。在文本层面,提示的重复使用程度适中:35.88%的唯一句子占了50%的验证片段,表明结构性集中主要由不均等的贡献者活动驱动,而非少数提示集的主导。这些结果提供了对快速扩展的低资源语音语料库的定量审计,并突显了改进数据集成熟度的实用优先事项,包括扩展验证能力以及更广泛的人口参与度。

英文摘要:

Large, openly licensed speech datasets are essential for building automatic speech recognition (ASR) systems, yet many widely spoken languages remain underrepresented in public resources. Pashto, spoken by more than 60 million people, has historically lacked large-scale openly licensed speech data suitable for modern ASR development. This paper presents a release-level analysis of the Pashto component of the Mozilla Common Voice corpus, focusing on version 24.0 (December 2025) and contextualizing trends across major releases. We document rapid growth from 1.49 recorded hours in mid-2023 to 2,768.7 total hours in 2025, including 975.89 validated hours available for supervised ASR training. Beyond scale, we analyze validation throughput, contributor participation inequality, demographic metadata completeness, and sentence-level concentration in the validated subset. We find that participation is extremely concentrated (Gini = 0.941), age representation is strongly skewed toward young adults, and 41.97\% of clips lack self-reported gender labels, limiting subgroup auditing based on metadata. At the textual level, prompt reuse is moderate: 35.88\% of unique sentences account for 50\% of validated clips, suggesting that structural concentration is driven primarily by uneven contributor activity rather than dominance of a small prompt set. These results provide a quantitative audit of a rapidly scaling low-resource speech corpus and highlight practical priorities for improving dataset maturity, including expanded validation capacity and broader demographic participation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑