arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.13166eess.AScs.AIcs.CY
收藏

FairLENS:评估执法语音识别中的公平性

FairLENS: Assessing Fairness in Law Enforcement Speech Recognition

Texas A&M University(德克萨斯农工大学);Axon Enterprise, Inc.(安讯企业公司);Rice University(莱斯大学)

Yicheng Wang, Mark Cusick, Mohamed Laila, Kate Puech, Zhengping Ji, Xia Hu, Michael Wilson, Noah Spitzer-Williams, Bryan Wheeler, Yasser Ibrahim

AI总结:

本文提出FairLENS公平性评估框架,含新评估方法与多场景数据集,对12个ASR模型评估发现模型间偏见差异及声学域变化引发新偏见,为用户选择模型提供指南。

摘要:

自动语音识别(ASR)技术已成为强大的工具,提升了执法场景中的效率。为了确保不同声学环境中人口群体的公平性,ASR引擎必须在现实环境中对多种说话者进行测试。然而,有信心地描述模型之间的公平性差异仍然是一个挑战。同时,大多数公开的ASR数据集不足以进行令人满意的公平性评估。为解决这些局限性,我们构建了FairLENS——一个系统性的公平性评估框架。我们提出了一种新颖且适应性强的评估方法,以检验不同模型之间的公平性差异。我们还收集了一个涵盖多种场景和人口维度的公平性评估数据集。利用该框架,我们对1个开源和11个商业可用的最先进ASR模型进行了公平性评估。我们的结果显示,某些模型比其他模型表现出更多的偏见,这为用户在特定现实场景中选择ASR模型时提供了公平性指南。我们进一步探讨了模型对特定人口群体的偏见,并观察到声学领域的变化可能导致新偏见的出现。

英文摘要:

Automatic speech recognition (ASR) techniques have become powerful tools, enhancing efficiency in law enforcement scenarios. To ensure fairness for demographic groups in different acoustic environments, ASR engines must be tested across a variety of speakers in realistic settings. However, describing the fairness discrepancies between models with confidence remains a challenge. Meanwhile, most public ASR datasets are insufficient to perform a satisfying fairness evaluation. To address the limitations, we built FairLENS - a systematic fairness evaluation framework. We propose a novel and adaptable evaluation method to examine the fairness disparity between different models. We also collected a fairness evaluation dataset covering multiple scenarios and demographic dimensions. Leveraging this framework, we conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models. Our results reveal that certain models exhibit more biases than others, serving as a fairness guideline for users to make informed choices when selecting ASR models for a given real-world scenario. We further explored model biases towards specific demographic groups and observed that shifts in the acoustic domain can lead to the emergence of new biases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑