arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.08274cs.CLcs.AI
收藏

在文档问答场景中,大型语言模型会有多大的幻觉?一项跨温度、上下文长度和硬件平台的1720亿token研究

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

Kamiwaza AI

JV Roig

AI总结:

研究通过大规模评估揭示了大型语言模型在文档问答中幻觉率与温度、上下文长度及硬件平台的关系,发现模型选择和温度设置显著影响准确性与伪造率。

摘要:

在提供文档基础上,大型语言模型实际会有多大的幻觉?尽管这个问题对于企业AI部署至关重要,但可靠的测量一直受到依赖静态数据集易受污染、基于LLM的裁判有已知偏见或评估尺度太小无法获得统计信心的基准的阻碍。我们通过RIKER,一种以真实事实为基础的评估方法,解决了这一缺口。在35个开放权重模型、三种上下文长度(32K、128K和200K tokens)、四种温度设置和三种硬件平台(NVIDIA H200、AMD MI300X和Intel Gaudi 3)上,我们进行了超过1720亿token的评估——比先前工作大一个数量级。我们的发现表明:(1)即使表现最好的模型也会以非平凡的比率生成答案——在32K时最佳为1.19%,顶级模型为5-7%——并且随着上下文长度增加,伪造率急剧上升,接近在128K时三倍,对于所有模型在200K时超过10%;(2)模型选择主导了所有其他因素,整体准确率跨度达72个百分点,模型家族预测伪造抵抗能力比模型大小更好;(3)温度效应是复杂的——T=0.0在大约60%的情况下产生最佳整体准确率,但更高温度对大多数模型减少伪造并大幅减少连贯性损失(无限生成循环),这在T=0.0时可达T=1.0时的48倍;(4)基础能力和伪造抵抗是不同的能力——擅长查找事实的模型可能仍然会伪造不存在的事实;(5)结果在不同硬件平台上一致,确认部署决策无需依赖硬件。

英文摘要:

How much do large language models actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to contamination, LLM-based judges with documented biases, or evaluation scales too small for statistical confidence. We address this gap using RIKER, a ground-truth-first evaluation methodology that enables deterministic scoring without human annotation. Across 35 open-weight models, three context lengths (32K, 128K, and 200K tokens), four temperature settings, and three hardware platforms (NVIDIA H200, AMD MI300X, and Intel Gaudi 3), we conducted over 172 billion tokens of evaluation - an order of magnitude beyond prior work. Our findings reveal that: (1) even the best-performing models fabricate answers at a non-trivial rate - 1.19% at best at 32K, with top-tier models at 5 - 7% - and fabrication rises steeply with context length, nearly tripling at 128K and exceeding 10% for all models at 200K; (2) model selection dominates all other factors, with overall accuracy spanning a 72-percentage-point range and model family predicting fabrication resistance better than model size; (3) temperature effects are nuanced - T=0.0 yields the best overall accuracy in roughly 60% of cases, but higher temperatures reduce fabrication for the majority of models and dramatically reduce coherence loss (infinite generation loops), which can reach 48x higher rates at T=0.0 versus T=1.0; (4) grounding ability and fabrication resistance are distinct capabilities - models that excel at finding facts may still fabricate facts that do not exist; and (5) results are consistent across hardware platforms, confirming that deployment decisions need not be hardware-dependent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑