arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~

AraDetox:多方言阿拉伯语解毒数据集

AraDetox: A Multi-Dialect Arabic Detoxification Dataset

Mo El-Haj

arXiv 2608.22894首次发表:更新:

AI 中文总结

本文提出多方言阿拉伯语解毒数据集AraDetox,基于GPT-5和Gemini 2.5 Flash生成海量解毒文本,经评估验证其可有效去除有害语言并保留原意,为相关研究提供公开资源。

AI 中文摘要

阿拉伯语有害语言检测已受到大量关注,但阿拉伯语文本解毒研究仍未得到充分探索。我们推出AraDetox,这是一个多方言阿拉伯语解毒数据集,包含10500条有害社交媒体帖子,以及使用GPT-5和Gemini 2.5 Flash生成的84000条解毒改写文本,覆盖现代标准阿拉伯语、海湾阿拉伯语、黎凡特阿拉伯语和埃及阿拉伯语。生成的输出通过人工评估以及对词汇变化、语义保留、情感和方言风格的自动分析进行评估。结果表明,解毒本质上是保留意义的改写任务:大量词汇和结构重构伴随着始终较高的语义相似度。人工评估确认成功去除有害语言,同时在很大程度上保留了原意。方言分析进一步表明,生成的变体与参考阿拉伯语方言语料库表现出可测量的风格一致性。与现有资源的比较突出了两种互补的解毒方法:最小编辑词汇替换和保留意义的重构。我们的研究结果表明,可通过大语言模型(LLM)辅助生成和人工验证构建大规模阿拉伯语解毒资源。该数据集已公开提供,可支持未来关于阿拉伯语解毒、安全文本生成和多方言阿拉伯语自然语言处理(NLP)的研究。

英文摘要

Arabic harmful-language detection has received considerable attention, yet Arabic text detoxification remains underexplored. We introduce AraDetox, a multi-dialect Arabic detoxification dataset comprising 10,500 harmful social-media posts and 84,000 detoxified rewrites generated using GPT-5 and Gemini 2.5 Flash across Modern Standard Arabic, Gulf, Levantine, and Egyptian Arabic. The generated outputs were assessed through human evaluation and automatic analyses of lexical change, semantic preservation, sentiment, and dialectal style. Results show that detoxification is primarily a meaning-preserving rewriting task: substantial lexical and structural reformulation is accompanied by consistently high semantic similarity. Human evaluation confirms successful harmful-language removal while largely preserving the original meaning. Dialectal analyses further indicate that the generated variants exhibit measurable stylistic alignment with reference Arabic dialect corpora. Comparison with existing resources highlights two complementary approaches to detoxification: minimal-edit lexical substitution and meaning-preserving reformulation. Our findings demonstrate that large-scale Arabic detoxification resources can be constructed through LLM-assisted generation and human verification. The dataset is publicly available at https://github.com/ArabicNLP-UK/AraDetox to support future research on Arabic detoxification, safe text generation, and multi-dialect Arabic NLP.

Comments15 pages, 6 figures

Journal refArabicNLP 2026 at EMNLP 2026 Conference

论文原文

arXiv 摘要页 · PDF 原文 · HTML 原文

↑