arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.15077cs.CLcs.AIcs.GT
收藏

利用大语言模型获取信息丰富的文本评价

Eliciting Informative Text Evaluations with Large Language Models

Peking University(北京大学);University of Michigan(密歇根大学)

Yuxuan Lu, Shengwei Xu, Yichi Zhang, Yuqing Kong, Grant Schoenebeck

AI总结:

本文提出GPPM和GSPPM两种机制,利用大语言模型作为预测器将同伴预测机制扩展到文本评价场景,并通过理论分析和Yelp、ICLR数据集实验验证其能激励真实高质量反馈并有效区分不同质量的评论。

摘要:

同伴预测机制以可证明的保证激励高质量反馈。然而,当前方法仅适用于相当简单的报告,例如多项选择或标量数值。我们旨在借助大语言模型的最新进展,将这些技术扩展到更广泛的基于文本的报告领域。这极大地提高了同伴预测机制的适用性,因为文本反馈在多种反馈渠道中是常态:同行评审、电子商务客户评价以及社交媒体评论。我们引入了两种机制:生成式同伴预测机制(GPPM)和生成式摘要同伴预测机制(GSPPM)。这些机制利用大语言模型作为预测器,将一个智能体的报告映射为其同伴报告的预测。理论上,我们证明当大语言模型的预测足够准确时,我们的机制能够以(近似)贝叶斯纳什均衡的方式激励高努力和真实报告。实证上,我们通过在两个真实数据集上进行的实验确认了我们机制的有效性:Yelp评论数据集和ICLR OpenReview数据集。我们强调的结果是,在ICLR数据集上,我们的机制能够在期望分数方面区分三个质量等级——人类撰写的评论、GPT-4生成的评论和GPT-3.5生成的评论。此外,GSPPM比GPPM更有效地惩罚大语言模型生成的评论。

英文摘要:

Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar numbers. We aim to broaden these techniques to the larger domain of text-based reports, drawing on the recent developments in large language models. This vastly increases the applicability of peer prediction mechanisms as textual feedback is the norm in a large variety of feedback channels: peer reviews, e-commerce customer reviews, and comments on social media. We introduce two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM). These mechanisms utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report. Theoretically, we show that when the LLM prediction is sufficiently accurate, our mechanisms can incentivize high effort and truth-telling as an (approximate) Bayesian Nash equilibrium. Empirically, we confirm the efficacy of our mechanisms through experiments conducted on two real datasets: the Yelp review dataset and the ICLR OpenReview dataset. We highlight the results that on the ICLR dataset, our mechanisms can differentiate three quality levels -- human-written reviews, GPT-4-generated reviews, and GPT-3.5-generated reviews in terms of expected scores. Additionally, GSPPM penalizes LLM-generated reviews more effectively than GPPM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑