用 Shona 俚语推进对话式 AI：面向数字包容的数据集与混合模型
Advancing Conversational AI with Shona Slang: A Dataset and Hybrid Model for Digital Inclusion
AI总结：
本研究针对非洲语言 Shona 语在 NLP 中缺乏非正式语料的不足，构建了 Shona-英语俚语数据集，并微调多语言 DistilBERT 模型以 96.4% 准确率实现意图识别，进而结合 RAG 构建出在文化相关性与用户参与度上表现更优的混合聊天机器人。
摘要：
非洲语言在自然语言处理（NLP）中仍然代表性不足，大多数语料库仅限于正式语域，未能捕捉日常交流的生动性。本研究通过引入一个从匿名社交媒体对话中精心策划的新型 Shona-英语俚语数据集，填补了津巴布韦和赞比亚使用的 Bantu 语——Shona 语的这一空白。该数据集针对意图、情感、对话行为、语码混合和语气进行了标注，并在 https://github.com/HappymoreMasoka/Working_with_shona-slang 上公开。我们微调了一个多语言 DistilBERT 分类器用于意图识别，达到了 96.4% 的准确率和 96.3% 的 F1 分数，并托管于 https://huggingface.co/HappymoreMasoka。该分类器被集成到一个混合聊天机器人中，该机器人结合了基于规则的响应和检索增强生成（RAG）来处理特定领域的查询，并通过一个协助 Pace University 准研究生获取项目信息的用例进行了演示。定性评估表明，该混合系统在文化相关性和用户参与度方面优于仅使用 RAG 的基线。通过发布数据集、模型和方法论，本研究推进了非洲语言的 NLP 资源，促进了包容且具有文化共鸣的对话式 AI。
英文摘要：
African languages remain underrepresented in natural language processing (NLP), with most corpora limited to formal registers that fail to capture the vibrancy of everyday communication. This work addresses this gap for Shona, a Bantu language spoken in Zimbabwe and Zambia, by introducing a novel Shona--English slang dataset curated from anonymized social media conversations. The dataset is annotated for intent, sentiment, dialogue acts, code-mixing, and tone, and is publicly available at https://github.com/HappymoreMasoka/Working_with_shona-slang. We fine-tuned a multilingual DistilBERT classifier for intent recognition, achieving 96.4\% accuracy and 96.3\% F1-score, hosted at https://huggingface.co/HappymoreMasoka. This classifier is integrated into a hybrid chatbot that combines rule-based responses with retrieval-augmented generation (RAG) to handle domain-specific queries, demonstrated through a use case assisting prospective students with graduate program information at Pace University. Qualitative evaluation shows the hybrid system outperforms a RAG-only baseline in cultural relevance and user engagement. By releasing the dataset, model, and methodology, this work advances NLP resources for African languages, promoting inclusive and culturally resonant conversational AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
