arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05201cs.CLcs.AI
收藏

采样温度对大型语言模型问题解决能力的影响

The Effect of Sampling Temperature on Problem Solving in Large Language Models

Johns Hopkins University(约翰斯·霍普金斯大学)

Matthew Renze, Erhan Guven

AI总结:

该研究通过九种LLM、五种提示工程技术和0.0至1.6的温度设置进行MCQA实验,发现0.0至1.0的温度变化对问题解决性能无显著影响,且结论具有跨模型、技术与领域的普适性。

摘要:

在本研究中,我们通过实证考察了采样温度对大型语言模型(LLM)在各类问题解决任务上表现的影响。我们通过从标准LLM基准中随机抽取问题,创建了一份多项选择问答(MCQA)测验。随后,我们使用九种流行的LLM,并结合五种提示工程技术来解决这些MCQA问题,同时将采样温度从0.0提高到1.6。尽管存在相反的传闻报告,但我们的实证结果表明,温度在0.0至1.0之间的变化对LLM在问题解决任务上的表现没有统计学上的显著影响。此外,这些结果似乎可以推广到不同的LLM、提示工程技术和问题领域。所有代码、数据和补充材料均可在GitHub获取:https://github.com/matthewrenze/jhu-llm-temperature

英文摘要:

In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑