arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.04667cs.CLcs.AIcs.LGcs.SE
收藏

“确定性”LLM设置的非确定性

Non-Determinism of "Deterministic" LLM Settings

Penn State University(宾夕法尼亚州立大学);Comcast AI Technologies(康卡斯特人工智能技术公司)
Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu,… 展开作者

Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, Breck Baldwin

AI总结:

本文研究了配置为确定性的5个LLM在8项任务上的非确定性问题,发现准确率差异高达15%,并引入TARr@N和TARa@N指标量化输出一致性。

摘要:

LLM(大语言模型)从业者普遍注意到,在预期为确定性的设置下,相同的输入可能会产生不同的输出。然而,据我们所知,这种现象有多普遍以及其对结果的影响如何,尚未得到系统性调查。我们研究了配置为确定性的5个LLM在零样本和少样本设置下,对8项常见任务进行10次运行时的非确定性。我们观察到,在自然发生的运行中,准确率差异高达15%,最佳可能性能与最差可能性能之间的差距高达70%。事实上,没有一个LLM能够在所有任务中始终提供可重复的准确率,更不用说相同的输出字符串了。与内部人士分享初步结果揭示,非确定性可能是通过输入缓冲区中的混合数据高效利用计算资源所必需的,因此这个问题短期内不会消失。为了更好地量化我们的观察结果,我们引入了专注于量化确定性的指标:TARr@N表示N次运行中原始输出的总一致率,TARa@N表示解析后答案的总一致率。我们的代码和数据公开于https://github.com/breckbaldwin/llm-stability。

英文摘要:

LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. Yet the questions of how pervasive this is, and with what impact on results, have not to our knowledge been systematically investigated. We investigate non-determinism in five LLMs configured to be deterministic when applied to eight common tasks in across 10 runs, in both zero-shot and few-shot settings. We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings. Sharing preliminary results with insiders has revealed that non-determinism perhaps essential to the efficient use of compute resources via co-mingled data in input buffers so this issue is not going away anytime soon. To better quantify our observations, we introduce metrics focused on quantifying determinism, TARr@N for the total agreement rate at N runs over raw output, and TARa@N for total agreement rate of parsed-out answers. Our code and data are publicly available at https://github.com/breckbaldwin/llm-stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑