自动化主题模型评估是否被破坏？：连贯性的不连贯性
Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
摘要：
主题模型评估，就像其他无监督方法的评估一样，可能会有争议。然而，该领域已经围绕主题连贯性的自动估计进行了合并，这依赖于参考语料库中单词共现的频率。根据这些指标，当代神经主题模型超过了经典模型。与此同时，主题模型评估存在验证差距：为经典模型开发的自动一致性尚未使用神经模型的人类实验进行验证。此外，主题建模文献的元分析揭示了自动主题建模基准测试中的实质性标准化差距。为了解决验证差距，我们比较了自动连贯性与两个最广泛接受的人类判断任务：主题评级和单词入侵。为了解决标准化的差距，我们在两个常用的数据集上系统地评估了一个占主导地位的经典模型和两个最先进的神经模型。自动评估宣布一个获胜的模型，而相应的人类评估没有，这就质疑了独立于人类判断的全自动评估的有效性。
英文摘要：
Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrences in a reference corpus. Contemporary neural topic models surpass classical ones according to these metrics. At the same time, topic model evaluation suffers from a validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, a meta-analysis of topic modeling literature reveals a substantial standardization gap in automated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets. Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
