AAVENUE：通过新型基准检测LLM在AAVE的NLU任务上的偏见
AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark
AI总结：
提出AAVENUE基准，用LLM翻译和少样本提示扩展VALUE，评估LLM在AAVE和SAE的NLU任务表现，发现LLM在SAE上表现更优，揭示偏见并强调包容性NLP需求。
摘要：
检测非裔美国人白话英语（AAVE）在自然语言理解（NLU）中的偏见对于开发包容性的自然语言处理（NLP）系统至关重要。为解决方言引起的性能差异，我们引入了AAVENUE（AAVE自然语言理解评估），一个用于评估大型语言模型（LLM）在AAVE和标准美国英语（SAE）的NLU任务上表现的基准。AAVENUE建立在并扩展了如VALUE等现有基准之上，用基于LLM翻译和少样本提示的更灵活方法取代了确定性的句法和形态转换，在从GLUE和SuperGLUE基准翻译关键任务时，提高了我们评估指标的性能。我们使用五个流行的LLM和包括流畅性、BARTScore、质量、连贯性和可理解性在内的全面指标集，比较了AAVENUE和VALUE的翻译。此外，我们招募了流利的AAVE使用者来验证我们翻译的真实性。我们的评估显示，LLM在SAE任务上的表现始终优于AAVE翻译版本，凸显了固有偏见，并强调了需要更具包容性的NLP模型。我们已在GitHub上开源了源代码，并在https://aavenuee.github.io创建了网站展示我们的工作。
英文摘要：
Detecting biases in natural language understanding (NLU) for African American Vernacular English (AAVE) is crucial to developing inclusive natural language processing (NLP) systems. To address dialect-induced performance discrepancies, we introduce AAVENUE ({AAVE} {N}atural Language {U}nderstanding {E}valuation), a benchmark for evaluating large language model (LLM) performance on NLU tasks in AAVE and Standard American English (SAE). AAVENUE builds upon and extends existing benchmarks like VALUE, replacing deterministic syntactic and morphological transformations with a more flexible methodology leveraging LLM-based translation with few-shot prompting, improving performance across our evaluation metrics when translating key tasks from the GLUE and SuperGLUE benchmarks. We compare AAVENUE and VALUE translations using five popular LLMs and a comprehensive set of metrics including fluency, BARTScore, quality, coherence, and understandability. Additionally, we recruit fluent AAVE speakers to validate our translations for authenticity. Our evaluations reveal that LLMs consistently perform better on SAE tasks than AAVE-translated versions, underscoring inherent biases and highlighting the need for more inclusive NLP models. We have open-sourced our source code on GitHub and created a website to showcase our work at https://aavenuee.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
