arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00623cs.HCcs.AI
收藏

“我不确定,但……”:探究大语言模型的不确定性表达对用户依赖与信任的影响

"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust

Princeton University(普林斯顿大学);Microsoft(微软公司)

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

AI总结:

本研究通过404人参与的预注册实验发现,LLM的第一人称自然语言不确定性表达能有效减少用户对错误答案的过度依赖、提升任务准确率,通用视角表达效果较弱且不显著,凸显LLM部署前用户测试的重要性。

摘要:

广泛部署的大语言模型(LLM)可能生成看似可信却错误的输出,可能误导那些将其输出视为正确结果而加以依赖的用户。为减少此类过度依赖,已有呼声要求LLM向终端用户传达其不确定性。然而,目前鲜有实证研究考察用户如何感知LLM的不确定性表达,以及如何据此采取行动。我们通过一项大规模、预注册的人类受试者实验(N=404)探究了这一问题,实验中参与者需要回答医学问题,部分参与者可使用一款虚构的搭载LLM的搜索引擎的回复,部分则不可。我们结合行为测量与自我报告测量,考察了不同的自然语言不确定性表达如何影响参与者的依赖程度、信任度以及整体任务表现。我们发现,第一人称表达(例如“我不确定,但……”)会降低参与者对系统的信心以及认同系统答案的倾向,同时提升参与者的准确率。一项探索性分析表明,这种准确率提升可归因于用户对错误答案的过度依赖有所减少(但并未完全消除)。虽然我们观察到从通用视角表达的不确定性(例如“目前尚不清楚,但……”)也有类似效果,但这些效果更弱,且不具备统计显著性。我们的研究结果表明,使用自然语言表达不确定性可能是减少用户对LLM过度依赖的有效方法,但具体使用的措辞至关重要。这凸显了在大规模部署LLM之前开展用户测试的重要性。

英文摘要:

Widely deployed large language models (LLMs) can produce convincing yet incorrect outputs, potentially misleading users who may rely on them as if they were correct. To reduce such overreliance, there have been calls for LLMs to communicate their uncertainty to end users. However, there has been little empirical work examining how users perceive and act upon LLMs' expressions of uncertainty. We explore this question through a large-scale, pre-registered, human-subject experiment (N=404) in which participants answer medical questions with or without access to responses from a fictional LLM-infused search engine. Using both behavioral and self-reported measures, we examine how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions (e.g., "I'm not sure, but...") decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. An exploratory analysis suggests that this increase can be attributed to reduced (but not fully eliminated) overreliance on incorrect answers. While we observe similar effects for uncertainty expressed from a general perspective (e.g., "It's not clear, but..."), these effects are weaker and not statistically significant. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters. This highlights the importance of user testing before deploying LLMs at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑