检测音译文本的语言
Detect Language of Transliterated Texts
摘要:
从其他语言到英语的非正式音译在社交媒体线程、即时消息和论坛中很普遍。如果不识别这种音译文本的语言,不讲该语言的用户就无法使用翻译工具理解其内容。我们提出了一个语言识别(LID)系统,与特征提取的方法,它可以检测到的音译文本的语言相当不错,即使有限的训练数据和计算资源。我们将单词标记为语音音节,并使用简单的长短期记忆(LSTM)网络架构来检测音译文本的语言。通过大量的实验,我们证明了音译词作为语音音节的标记化有效地代表了它们的因果语音模式。因此,语音音节标记化使得更简单的模型架构更容易学习识别任何语言的特征模式。
英文摘要:
Informal transliteration from other languages to English is prevalent in social media threads, instant messaging, and discussion forums. Without identifying the language of such transliterated text, users who do not speak that language cannot understand its content using translation tools. We propose a Language Identification (LID) system, with an approach for feature extraction, which can detect the language of transliterated texts reasonably well even with limited training data and computational resources. We tokenize the words into phonetic syllables and use a simple Long Short-term Memory (LSTM) network architecture to detect the language of transliterated texts. With intensive experiments, we show that the tokenization of transliterated words as phonetic syllables effectively represents their causal sound patterns. Phonetic syllable tokenization, therefore, makes it easier for even simpler model architectures to learn the characteristic patterns to identify any language.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。