arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1804.00804cs.CL
收藏

语码混合社交媒体文本中词汇变体的自动归一化

Automatic Normalization of Word Variations in Code-Mixed Social Media Text

Language Technologies Research Centre (LTRC)(语言技术研究中心);Kohli Center on Intelligent Systems (KCIS)(科利智能系统中心);International Institute of Information Technology, Hyderabad(海得拉巴国际信息技术学院)

Rajat Singh, Nurendra Choudhary, Manish Shrivastava

AI总结:

针对语码混合社交媒体文本的拼写变体问题,提出利用词分布式表示无监督捕捉同上下文词汇变体的归一化方法,可提升词性标注与情感分析任务性能。

摘要:

Twitter、Facebook等社交媒体平台在多语言社会中日益普及,这一趋势催生了南亚语言与英语的混合词。这种由多种语言融合形成的语码混合数据,近期在研究界被广泛用于各类NLP任务。语码混合数据存在语法错误、拼写变体等异常情况。本文利用词汇的上下文特性——在大量含噪社交媒体文本中,同一词汇的不同拼写变体共享相似上下文。我们采用无监督方式,借助词的分布式表示捕捉属于同一上下文的不同词汇变体。实验表明,基于本文方法对语码混合数据集进行预处理,可提升当前最优的词性标注(POS-tagging)与情感分析任务的性能。

英文摘要:

Social media platforms such as Twitter and Facebook are becoming popular in multilingual societies. This trend induces portmanteau of South Asian languages with English. The blend of multiple languages as code-mixed data has recently become popular in research communities for various NLP tasks. Code-mixed data consist of anomalies such as grammatical errors and spelling variations. In this paper, we leverage the contextual property of words where the different spelling variation of words share similar context in a large noisy social media text. We capture different variations of words belonging to same context in an unsupervised manner using distributed representations of words. Our experiments reveal that preprocessing of the code-mixed dataset based on our approach improves the performance in state-of-the-art part-of-speech tagging (POS-tagging) and sentiment analysis tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑