arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.16710cs.CEcs.CL
收藏

超越图灵测试:GPT-4 能否影响专家决策?

Beyond Turing Test: Can GPT-4 Sway Experts' Decisions?

The University of Tokyo(东京大学);National Institute of Advanced Industrial Science and Technology(独立行政法人产业技术综合研究所)

Takehiro Takayanagi, Hiroya Takamura, Kiyoshi Izumi, Chung-Chi Chen

AI总结:

本文聚焦后图灵时代LLM评估新方向,探究GPT-4生成文本对业余及专家决策的影响,发现其具说服力且真实世界受众反应评估与多维度指标高度相关,还揭示相关潜力风险并发布数据集。

摘要:

在后图灵时代,评估大型语言模型(LLMs)需基于读者反应评估生成文本,而非仅关注其与人类生成内容的不可区分性。本文探究LLM生成文本如何影响读者决策,聚焦业余和专家受众。研究发现,GPT-4能生成具有说服力的分析,影响业余者和专业人士的决策。此外,我们从语法、说服力、逻辑连贯性和有用性方面评估生成文本。结果显示,通过受众反应进行的真实世界评估与当前生成模型常用的多维度评估指标高度相关。总体而言,本文展示了使用生成文本影响人类决策的潜力与风险,并指出评估生成文本的新方向,即利用读者的反应和决策。我们已发布数据集以助力未来研究。

英文摘要:

In the post-Turing era, evaluating large language models (LLMs) involves assessing generated text based on readers' reactions rather than merely its indistinguishability from human-produced content. This paper explores how LLM-generated text impacts readers' decisions, focusing on both amateur and expert audiences. Our findings indicate that GPT-4 can generate persuasive analyses affecting the decisions of both amateurs and professionals. Furthermore, we evaluate the generated text from the aspects of grammar, convincingness, logical coherence, and usefulness. The results highlight a high correlation between real-world evaluation through audience reactions and the current multi-dimensional evaluators commonly used for generative models. Overall, this paper shows the potential and risk of using generated text to sway human decisions and also points out a new direction for evaluating generated text, i.e., leveraging the reactions and decisions of readers. We release our dataset to assist future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑