相同提示,不同结果:评估由LLM进行的数据分析的可重复性
Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs
AI总结:
本研究评估了LLM在相同任务和设置下进行数据分析的可重复性,发现结果存在显著差异,强调了多次独立运行和结果分布分析的必要性。
摘要:
我们系统地评估了由大型语言模型(LLMs)进行的数据分析的可重复性。我们评估了两种提示策略、六种模型和四种温度设置,每个配置进行十次独立执行,共计480次尝试。我们评估了每次尝试的完成情况、一致性、有效性及一致性,并发现即使在一致的配置下,分析结果也存在显著差异。这表明,与人类数据处理类似,由LLM进行的数据分析在给定相同任务、数据和设置的情况下也可能存在差异。我们的结果表明,如果使用LLM进行数据处理,则应多次独立运行并考虑结果的分布。
英文摘要:
We systematically evaluate the reproducibility of data analysis conducted by Large Language Models (LLMs). We evaluate two prompting strategies, six models, and four temperature settings, with ten independent executions per configuration, yielding 480 total attempts. We assess the completion, concordance, validity, and consistency of each attempt and find considerable variation in the analytical results even for consistent configurations. This suggests, as with human data analysis, the data analysis conducted by LLMs can vary, even given the same task, data, and settings. Our results mean that if an LLM is being used to conduct data analysis, then it should be run multiple times independently and the distribution of results considered.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。