Vi-Mistral-X：基于高级持续预训练构建的越南语语言模型
Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training
AI总结：
本文提出专为越南语设计的语言模型vi-mistral-x，基于Mistral架构采用分组查询与滑动窗口注意力进行持续预训练，在文本分类、问答及生成等基准上显著优于现有越南语模型。
摘要：
大型语言模型（LLMs）的发展显著改变了自然语言处理领域，尽管以英语为中心的模型导致包括越南语在内的特定语言存在明显的研究空白。为了解决这个问题，本文提出了vi-mistral-x，这是一种专为越南语设计的创新型大型语言模型。它采用基于Mistral架构的独特持续预训练方法，结合了分组查询注意力和滑动窗口注意力技术。该模型vi-mistral-X在改进越南语的理解和生成方面迈出了重要一步。它引入了专门针对越南语适配的额外持续预训练阶段，增强了模型理解复杂语言细微差别以及生成准确、具备上下文感知的越南语文本的能力。通过在多个基准上的全面测试，vi-mistral-x在文本分类、问答和文本生成等几个关键领域表现出优于现有越南语LLMs的性能。特别是在越南语多任务语言理解（VMLU）基准中，vi-mistral-x树立了新标准，显著优于其他可用模型。本文强调了持续预训练在推进特定语言LLMs方面的关键作用，并为多语言模型的开发开辟了新途径。我们希望vi-mistral-x不仅是处理越南语的重要资产，也能鼓励为代表性不足的语言创建大型语言模型方面取得更多进展。
英文摘要：
The advancement of Large Language Models (LLMs) has significantly transformed the field of natural language processing, although the focus on English-centric models has created a noticeable research gap for specific languages, including Vietnamese. To address this issue, this paper presents vi-mistral-x, an innovative Large Language Model designed expressly for the Vietnamese language. It utilizes a unique method of continual pre-training, based on the Mistral architecture, which incorporates grouped-query attention and sliding window attention techniques. This model, vi-Mistral-X, marks a significant step forward in improving the understanding and generation of the Vietnamese language. It introduces an additional phase of continual pre-training, specifically adapted for Vietnamese, enhancing the model's capability in understanding complex language nuances and generating accurate, context-aware Vietnamese text. Through comprehensive testing on various benchmarks, vi-mistral-x has shown to outperform existing Vietnamese LLMs in several key areas, including text classification, question answering, and text generation. Particularly, in the Vietnamese Multitask Language Understanding (VMLU) benchmark, vi-mistral-x sets a new standard, outperforming other available models significantly. This paper highlights the critical role of continual pre-training in advancing language-specific LLMs and opens new avenues for the development of multilingual models. We aim for vi-mistral-x to not just be an important asset for processing the Vietnamese language but also to encourage more advancements in creating large language models for languages that are less represented.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
