arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09008cs.CL
收藏

LLM Reading Tea Leaves：利用大语言模型自动评估主题模型

LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models

Monash University（莫纳什大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61）；VinUniversity（维恩大学）

Xiaohao Yang, He Zhao, Dinh Phung, Wray Buntine, Lan Du

备注：
Forthcoming in Transactions of the Association for Computational Linguistics (TACL) published by MIT Press
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有主题模型评估方法可比性差、评估维度单一的问题，提出联合考量文档表示与主题语义质量的WALM评估方法，经实验验证其与人类判断一致，可作为现有方法的补充。

摘要：

主题建模一直是无监督文本分析中广泛使用的工具。然而，对主题模型进行全面评估仍然颇具挑战。现有评估方法要么在不同模型间可比性较差（例如困惑度），要么每次仅聚焦模型的某一个特定方面（例如主题质量或文档表示质量），不足以反映模型的整体性能。本文提出WALM（词与语言模型一致性，Word Agreement with Language Model），这是一种全新的主题建模评估方法，它借助大语言模型（LLMs）的能力，同时考量文档表示和主题的语义质量。通过涵盖不同类型主题模型的大量实验，WALM被证明与人类判断相一致，可作为现有评估方法的补充，为主题建模领域带来新的研究视角。我们的软件包可在https://github.com/Xiaohao-Yang/Topic_Model_Evaluation 获取。

英文摘要：

Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Word Agreement with Language Model), a new evaluation method for topic modeling that considers the semantic quality of document representations and topics in a joint manner, leveraging the power of Large Language Models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package is available at https://github.com/Xiaohao-Yang/Topic_Model_Evaluation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑