非正式罗马化解读的语音和视觉先验
Phonetic and Visual Priors for Decipherment of Informal Romanization
摘要:
非正式罗马化(英语:Informal Romanization)是人类在非正式数字通信中使用的一种特殊过程,用于将非拉丁字母语言编码为常见键盘上的拉丁字符集。字符替换的选择在用户之间是不同的,但是已经被证明是由在各种语言中观察到的相同的主要原则所支配的-即,字符对通常通过语音或视觉相似性来关联。我们提出了一个噪声通道WFST级联模型,用于以无监督的方式从观察到的罗马化文本中破译原始的非拉丁字母。我们直接在两种语言的罗马化数据上训练我们的模型:埃及语、阿拉伯语和俄语。我们证明,通过语音和视觉先验的字符映射添加归纳偏见,大大提高了模型在两种语言上的性能,产生的结果更接近监督的天际线。最后,我们介绍了一个新的数据集的罗马化俄罗斯,从俄罗斯的社交网络网站收集和部分注释我们的实验。
英文摘要:
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards. Character substitution choices differ between users but have been shown to be governed by the same main principles observed across a variety of languages---namely, character pairs are often associated through phonetic or visual similarity. We propose a noisy-channel WFST cascade model for deciphering the original non-Latin script from observed romanized text in an unsupervised fashion. We train our model directly on romanized data from two languages: Egyptian Arabic and Russian. We demonstrate that adding inductive bias through phonetic and visual priors on character mappings substantially improves the model's performance on both languages, yielding results much closer to the supervised skyline. Finally, we introduce a new dataset of romanized Russian, collected from a Russian social network website and partially annotated for our experiments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。