arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.19934cs.CLcs.AI
收藏

量化大型语言模型中的非确定性漂移

Quantifying non deterministic drift in large language models

Claire Nicholson

AI总结:

本研究通过实验量化大型语言模型在无操作员条件下的非确定性漂移,揭示了不同模型大小和部署类型下的输出变化模式,并探讨了语义方法在评估漂移缓解技术中的应用。

摘要:

大型语言模型(LLMs)被广泛用于从摘要到决策支持在内的各种任务。在实践中,即使温度和其他解码参数固定,相同的提示也不总是产生相同的输出。在本工作中,我们通过重复运行实验来经验性地量化基线行为漂移,定义为在无操作员条件下多次发出相同提示时观察到的输出变化。我们评估了两个公开可访问的模型,gpt-4o-mini和llama3.1-8b,在五个提示类别上使用精确重复、扰动输入和重用模式,在温度为0.0和0.7的情况下进行评估。通过使用唯一的输出分数、词法相似性和词数统计来测量漂移,可以实现跨模型、提示模式和部署类型的直接比较。结果表明,即使在温度为0.0时,非确定性仍然存在,不同模型大小、部署和提示类型表现出不同的变化模式。我们将这些发现置于现有概念漂移、行为漂移和基础设施引起的非确定性研究中,讨论了词法指标的局限性,并突出了新兴的语义方法。通过在没有稳定技术的情况下建立系统经验基线,本研究为评估未来漂移缓解和控制方法提供了参考点。

英文摘要:

Large language models (LLMs) are widely used for tasks ranging from summarisation to decision support. In practice, identical prompts do not always produce identical outputs, even when temperature and other decoding parameters are fixed. In this work, we conduct repeated-run experiments to empirically quantify baseline behavioural drift, defined as output variability observed when the same prompt is issued multiple times under operator-free conditions. We evaluate two publicly accessible models, gpt-4o-mini and llama3.1-8b, across five prompt categories using exact repeats, perturbed inputs, and reuse modes at temperatures of 0.0 and 0.7. Drift is measured using unique output fractions, lexical similarity, and word count statistics, enabling direct comparison across models, prompting modes, and deployment types. The results show that nondeterminism persists even at temperature 0.0, with distinct variability patterns by model size, deployment, and prompt type. We situate these findings within existing work on concept drift, behavioural drift, and infrastructure-induced nondeterminism, discuss the limitations of lexical metrics, and highlight emerging semantic approaches. By establishing a systematic empirical baseline in the absence of stabilisation techniques, this study provides a reference point for evaluating future drift mitigation and control methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑