arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.05651cs.CLcs.DBcs.LG
收藏

语义标记的深层模型还是简单模型?这取决于你的数据[实验]

Deep or Simple Models for Semantic Tagging? It Depends on your Data [Experiments]

Megagon Labs(Megagon Labs(Megagon实验室))

Jinfeng Li, Yuliang Li, Xiaolan Wang, Wang-Chiew Tan

摘要:

语义标记在文本挖掘中有着广泛的应用,它预测给定的文本是否传达了给定的语义标记的含义。语义标记的问题在很大程度上是通过监督学习来解决的,而今天,深度学习模型被广泛认为更适合语义标记。然而,没有全面的研究支持流行的信念。实践者经常必须为每个语义标记任务训练不同类型的模型以识别最佳模型。这个过程既昂贵又低效。我们开始了一项系统的研究,以调查以下问题:深度模型是所有语义标记任务的最佳执行模型吗?为了回答这个问题,我们将深度模型与具有不同特征的数据集上的“简单模型”进行比较。具体来说,我们选择了三种流行的深度模型(即CNN,LSTM和BERT)和两种简单模型(即LR和SVM),并比较了它们在21个数据集上的语义标记任务的性能。结果表明,数据集的大小,标签比例和标签清洁度显着影响语义标注的质量。简单模型在大型数据集上实现了与深度模型相似的标记质量,但简单模型的运行时间要短得多。此外,当目标数据集显示更差的标签清洁度和/或更严重的不平衡时,简单模型可以实现比深度模型更好的标记质量。基于这些发现,我们的研究可以系统地指导从业者选择正确的学习模式,他们的语义标记任务。

英文摘要:

Semantic tagging, which has extensive applications in text mining, predicts whether a given piece of text conveys the meaning of a given semantic tag. The problem of semantic tagging is largely solved with supervised learning and today, deep learning models are widely perceived to be better for semantic tagging. However, there is no comprehensive study supporting the popular belief. Practitioners often have to train different types of models for each semantic tagging task to identify the best model. This process is both expensive and inefficient. We embark on a systematic study to investigate the following question: Are deep models the best performing model for all semantic tagging tasks? To answer this question, we compare deep models against "simple models" over datasets with varying characteristics. Specifically, we select three prevalent deep models (i.e. CNN, LSTM, and BERT) and two simple models (i.e. LR and SVM), and compare their performance on the semantic tagging task over 21 datasets. Results show that the size, the label ratio, and the label cleanliness of a dataset significantly impact the quality of semantic tagging. Simple models achieve similar tagging quality to deep models on large datasets, but the runtime of simple models is much shorter. Moreover, simple models can achieve better tagging quality than deep models when targeting datasets show worse label cleanliness and/or more severe imbalance. Based on these findings, our study can systematically guide practitioners in selecting the right learning model for their semantic tagging task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑