arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.08091cs.CLcs.AIcs.IRcs.LG
收藏

推文情绪量化:一个实验性的重新评估

Tweet Sentiment Quantification: An Experimental Re-Evaluation

Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche(意大利国家研究委员会信息科学与技术研究所)

Alejandro Moreo, Fabrizio Sebastiani

摘要:

情感量化是通过监督学习来训练未标记文本样本中情感相关类(例如\textsf{Positive}、\textsf{Neutral}、\textsf{Negative})的相对频率(也称为“普遍性”)的估计器的任务。当这些文本是推文时,这项任务尤其重要,因为对Twitter数据进行的大多数情感分类工作的最终目标实际上是量化(而不是对单个推文进行分类)。众所周知,通过“分类和计数”(即,通过标准分类器对所有未标记的项目进行分类并对已分配给给定类别的项目进行计数)在准确性方面不是最佳的,并且存在更准确的量化方法。Gao和Reintiani(2016)对推文情感量化任务的量化方法进行了系统的比较。事后看来,我们观察到,在这项工作中遵循的实验方案是薄弱的,因此从结果中得出的结论的可靠性是值得怀疑的。我们现在在完全相同的数据集上重新评估这些量化方法(加上一些更现代的方法),这次遵循的是一个现在统一的、更强大的实验方案(其中还包括模拟测试数据中存在的与训练集非常不同的类流行率值)。该实验方案(即使不计算新添加的方法)涉及的实验数量是原始研究的5,775倍。我们的实验结果与Gao和Rumatiani获得的结果有很大的不同,它们提供了一个不同的,更坚实的理解不同情感量化方法的相对优势和劣势。

英文摘要:

Sentiment quantification is the task of training, by means of supervised learning, estimators of the relative frequency (also called ``prevalence'') of sentiment-related classes (such as \textsf{Positive}, \textsf{Neutral}, \textsf{Negative}) in a sample of unlabelled texts. This task is especially important when these texts are tweets, since the final goal of most sentiment classification efforts carried out on Twitter data is actually quantification (and not the classification of individual tweets). It is well-known that solving quantification by means of ``classify and count'' (i.e., by classifying all unlabelled items by means of a standard classifier and counting the items that have been assigned to a given class) is less than optimal in terms of accuracy, and that more accurate quantification methods exist. Gao and Sebastiani (2016) carried out a systematic comparison of quantification methods on the task of tweet sentiment quantification. In hindsight, we observe that the experimental protocol followed in that work was weak, and that the reliability of the conclusions that were drawn from the results is thus questionable. We now re-evaluate those quantification methods (plus a few more modern ones) on exactly the same same datasets, this time following a now consolidated and much more robust experimental protocol (which also involves simulating the presence, in the test data, of class prevalence values very different from those of the training set). This experimental protocol (even without counting the newly added methods) involves a number of experiments 5,775 times larger than that of the original study. The results of our experiments are dramatically different from those obtained by Gao and Sebastiani, and they provide a different, much more solid understanding of the relative strengths and weaknesses of different sentiment quantification methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑