arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.01308cs.CLcs.AIcs.LG
收藏

VBART:土耳其语大语言模型

VBART: The Turkish LLM

VNGRS-AI;YTÜ Teknopark(伊斯坦布尔理工大学科技园)

Meliksah Turker, Mehmet Erdi Ari, Aydin Han

AI总结:

本文提出VBART,首个土耳其语序列到序列大语言模型,基于BART/mBART思想,在摘要、问答等任务上超越SOTA,性能优于多语言模型3倍,并公开模型与语料。

摘要:

我们提出了VBART,这是首个从零开始在大型语料库上预训练的土耳其语序列到序列大语言模型(LLMs)。VBART是基于BART和mBART模型中的优秀思想构建的紧凑型大语言模型,提供Large和XLarge两种规模。经过微调的VBART模型在抽象文本摘要、标题生成、文本改写、问答和问题生成任务上超越了先前的最先进结果。它们允许针对未来的文本生成任务和数据集进行微调,为土耳其语自然语言处理(NLP)研究开辟了新路径。我们的工作表明,拥有一个针对土耳其语的预训练大语言模型,其性能比多语言模型高出最多3倍,改善了现有结果,并为训练和推理提供了高效模型。此外,我们展示了我们的单语分词器比多语言分词器高效最多11倍。最后但同样重要的是,我们介绍了一种扩展现有预训练大语言模型的方法,并质疑Chinchilla缩放定律对序列到序列掩码语言模型的相关性。我们的微调模型、分词器和清理后的135 GB vngrs-web-corpus已在huggingface.co/vngrs-ai上公开提供。

英文摘要:

We present VBART, the first Turkish sequence-to-sequence Large Language Models (LLMs) pre-trained on a large corpus from scratch. VBART are compact LLMs based on good ideas leveraged from BART and mBART models and come in two sizes, Large and XLarge. Fine-tuned VBART models surpass the prior state-of-the-art results in abstractive text summarization, title generation, text paraphrasing, question answering and question generation tasks. They allow fine-tuning for future text generation tasks and datasets, carving a new path for Turkish Natural Language Processing (NLP) research. Our work shows that having a pre-trained LLM for Turkish outperforms up to 3x multilingual models, improving existing results and providing efficient models for training and inference. Moreover, we show that our monolingual tokenizer is up to 11x more efficient than multilingual tokenizers. Last but not least, we introduce a method to enlarge an existing pre-trained LLM and question the relevancy of Chinchilla Scaling Law to sequence-to-sequence masked language models. Our fine-tuned models, tokenizer and cleaned vngrs-web-corpus of 135 GB are publicly available at huggingface.co/vngrs-ai.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑