arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.14969cs.CLcs.AIcs.LG
收藏

Masked Part-Of-Speech Model:长上下文建模是否有助于无监督POS标记?

Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?

University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Xiang Zhou, Shiyue Zhang, Mohit Bansal

摘要:

先前的词性(POS)归纳模型通常假设某些独立性假设(例如,马尔可夫,单向,局部依赖),在真正的语言中不成立。例如,主谓一致既可以是长期的,也可以是双向的。为了方便灵活的依赖建模,我们提出了一个掩蔽部分的语音模型(MPoSM),最近成功的掩蔽语言模型(MLM)的启发。MPoSM可以对任意标签依赖进行建模,并通过掩码POS重构的目标进行POS归纳。我们在英语Penn WSJ数据集和包含10种不同语言的通用树库上都取得了有竞争力的结果。虽然对长期依赖性进行建模应该有助于完成这项任务,但我们的消融研究显示了不同语言的混合趋势。为了更好地理解这种现象,我们设计了一个新的合成实验,可以专门诊断模型学习标签一致性的能力。令人惊讶的是,我们发现即使是强大的基线也无法在非常简单的设置中一致地解决这个问题:相邻单词之间的一致性。尽管如此,MPoSM总体上实现了更好的性能。最后,我们进行了详细的错误分析,以揭示其他剩余的挑战。我们的代码可在https://github.com/owenzx/MPoSM上获得

英文摘要:

Previous Part-Of-Speech (POS) induction models usually assume certain independence assumptions (e.g., Markov, unidirectional, local dependency) that do not hold in real languages. For example, the subject-verb agreement can be both long-term and bidirectional. To facilitate flexible dependency modeling, we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent success of Masked Language Models (MLM). MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. We achieve competitive results on both the English Penn WSJ dataset as well as the universal treebank containing 10 diverse languages. Though modeling the long-term dependency should ideally help this task, our ablation study shows mixed trends in different languages. To better understand this phenomenon, we design a novel synthetic experiment that can specifically diagnose the model's ability to learn tag agreement. Surprisingly, we find that even strong baselines fail to solve this problem consistently in a very simplified setting: the agreement between adjacent words. Nonetheless, MPoSM achieves overall better performance. Lastly, we conduct a detailed error analysis to shed light on other remaining challenges. Our code is available at https://github.com/owenzx/MPoSM

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑