Alif：通过多语言合成数据蒸馏推进乌尔都语大语言模型
Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation
AI总结：
针对乌尔都语低资源挑战，提出Alif-1.0-8B-Instruct模型，利用改进的self-instruct技术生成多语言合成数据，在低于100美元预算下超越多个领先多语言模型，实现高效且文化对齐的LLM开发。
摘要：
为乌尔都语等低资源语言开发高性能大语言模型（LLMs）面临诸多挑战，包括高质量数据集稀缺、多语言不一致以及安全问题。现有的多语言大语言模型通常通过翻译大量可用数据来解决这些问题，然而，此类翻译往往缺乏质量和文化细微差别，同时数据整理和训练成本高昂。为解决这些问题，我们提出了Alif-1.0-8B-Instruct，一个多语言乌尔都语-英语模型，采用独特方法应对这些挑战。我们使用改进的self-instruct技术，在高质量多语言合成数据集（Urdu-Instruct）上训练该模型。通过为每个任务使用独特的提示和种子值以及全局任务池，该数据集融入了乌尔都语原生的思维链推理、双语翻译、文化相关性和伦理安全对齐。该技术显著增强了Alif-1.0-8B-Instruct模型对乌尔都语特定任务的理解能力。因此，基于预训练的Llama-3.1-8B构建的Alif-1.0-8B-Instruct，在乌尔都语特定任务上表现出优于Llama-3.1-8B-Instruct的性能。它还超越了领先的多语言大语言模型，包括Mistral-7B-Instruct-v0.3、Qwen-2.5-7B-Instruct和Cohere-Aya-Expanse-8B，且训练预算低于100美元。我们的结果表明，使用我们改进的self-instruct方法，可以高效地开发高性能且文化对齐的低资源语言大语言模型。所有数据集、模型和代码均公开提供：https://github.com/traversaal-ai/alif-urdu-llm。
英文摘要：
Developing a high-performing large language models (LLMs) for low-resource languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality datasets, multilingual inconsistencies, and safety concerns. Existing multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique prompts and seed values for each task along with a global task pool, this dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under $100. Our results demonstrate that high-performance and low-resource language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach. All datasets, models, and code are publicly available at: https://github.com/traversaal-ai/alif-urdu-llm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
