arXiv 2511.227692025-11-27cs.CL

构建罗马化印地语和孟加拉语的模型：数据集创建与多语言大语言模型整合

Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration

Department of EECS, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA（电子工程与科学系，埃姆布里-里德尔航空大学，日落海滩，佛罗里达州，美国）；NLP Engineering Team, Hishab Singapore Pte. Ltd, Singapore（自然语言处理工程团队，Hishab新加坡私人有限公司，新加坡）；Department of Computer Information Systems, Texas A&M University - Central Texas, Texas, USA（计算机信息系统系，德克萨斯A&M大学-中央德克萨斯分校，德克萨斯州，美国）

Kanchon Gharami, Quazi Sarwar Muhtaseem, Deepti Gupta, Lavanya Elluri, Shafika Showkat Moni

备注：: Proceedings of the 8th Workshop on Big Data for Cybersecurity (BigCyber)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种新的转写数据集和定制多语言模型，用于提高罗马化印地语和孟加拉语的转写效果。

摘要：

开发稳健的转写技术以提高将罗马化文字转换为原生文字的效果对于自然语言处理任务至关重要，包括情感分析、语音识别、信息检索和智能个人助理。尽管取得了显著进展，最先进的多语言模型在处理罗马化文字时仍面临挑战，因为罗马字母被用来表示不同语言的发音结构。在南亚背景下，罗马化文字在社交媒体和数字通信平台上广泛用于印欧语系语言，这种使用继续对先进多语言模型构成重大挑战。虽然为印欧语系语言提供了有限的转写数据集和模型，但它们通常缺乏足够的发音和拼写变化多样性、足够的代码混合数据用于大语言模型（LLM）训练以及低资源适应性。为了解决这一研究空白，我们引入了一个新的转写数据集，用于两种流行的印欧语系语言，印地语和孟加拉语，它们分别被列为全球第三和第七大语言。我们的数据集包含近180万个印地语和100万个孟加拉语转写对。此外，我们还基于Marian架构使用开发的数据集预训练了一个定制的多语言seq2seq LLM。实验结果表明，与现有相关模型相比，在BLEU和CER指标上取得了显著改进。

英文摘要：

The development of robust transliteration techniques to enhance the effectiveness of transforming Romanized scripts into native scripts is crucial for Natural Language Processing tasks, including sentiment analysis, speech recognition, information retrieval, and intelligent personal assistants. Despite significant advancements, state-of-the-art multilingual models still face challenges in handling Romanized script, where the Roman alphabet is adopted to represent the phonetic structure of diverse languages. Within the South Asian context, where the use of Romanized script for Indo-Aryan languages is widespread across social media and digital communication platforms, such usage continues to pose significant challenges for cutting-edge multilingual models. While a limited number of transliteration datasets and models are available for Indo-Aryan languages, they generally lack sufficient diversity in pronunciation and spelling variations, adequate code-mixed data for large language model (LLM) training, and low-resource adaptation. To address this research gap, we introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali, which are ranked as the 3rd and 7th most spoken languages worldwide. Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs. In addition to that, we pre-train a custom multilingual seq2seq LLM based on Marian architecture using the developed dataset. Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。