arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.03378cs.CL
收藏

面向英语隐式话语关系分类的Transformer模型并列比较

A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

University of Pennsylvania(宾夕法尼亚大学);LXPER AI Research(LXPER人工智能研究院)

Bruce W. Lee, BongSeok Yang, Jason Hyung-Jong Lee

AI总结:

本文在PDTB-3上对七个预训练语言模型进行隐式话语关系分类的并列比较,将SOTA提升至0.671 ACC,并发现句子级预训练目标通常不如MLM加全注意力机制有效。

摘要:

尽管话语解析能够助力多个自然语言处理领域,但在隐式话语关系分类任务上,尚未开展过广泛的语言模型搜索。这一现状阻碍了研究人员在话语分析中充分利用公开可用的模型。本工作对七个预训练语言模型进行了直接的、经过微调的话语性能比较。我们使用PDTB-3这一流行的话语关系标注数据集。通过模型搜索,我们将最优结果(SOTA)提升至0.671 ACC,并获得了新的观察发现。其中一些发现与先前报道的结论(Shi and Demberg, 2019b)相反,即句子级预训练目标(NSP、SBO、SOP)通常无法为隐式话语关系分类产生性能最佳的模型。与直觉相反的是,具有掩码语言模型(MLM)和全注意力机制的相近规模预训练语言模型(PLM)带来了更好的性能。

英文摘要:

Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification. This hinders researchers from fully utilizing public-available models in discourse analysis. This work is a straightforward, fine-tuned discourse performance comparison of seven pre-trained language models. We use PDTB-3, a popular discourse relation annotated dataset. Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations. Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best performing model for implicit discourse relation classification. Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑