幻觉存在于方差中
Hallucinations Live in Variance
AI总结:
研究提出语义稳定性(SS)作为衡量模型方差驱动不可靠性的诊断工具,通过同义词一致性(PC@k)评估模型在不同稀疏度下的稳定性,发现稀疏度增加可显著提升一致性。
摘要:
基准测试衡量模型是否正确,但不衡量模型是否可靠。这种区分在单次推理中大多是学术性的,但在代理AI系统中变得至关重要,因为一个重新表述的提示可能触发多步骤执行中的级联故障。然而,这种不稳定性并未被现有评估所捕捉。幻觉存在于方差中:当语义等价的提示激活不一致的内部路径时,会产生分歧的输出。一致但错误的输出反映偏见或知识缺失;自信猜测反映校准失败。在这定义下,两者都不构成幻觉。当错误由方差主导时,减少冗余路径可以提高可靠性而不增加知识。我们通过语义稳定性(SS)正式化这一概念,通过同义词一致性(PC@k)测量:生成k个同义词,贪婪解码每个,计算模式一致率。SS是用于方差驱动不可靠性的诊断工具,而不是用于提高正确性的方法。我们显示,密集的Qwen3-0.6B仅在23.8%的时间与自身一致;在32%稀疏度下,一致性跃升至55.9%。相图揭示了方差减少超过偏见积累的黄金点,以及稳定性崩溃到错误答案的区域。
英文摘要:
Benchmarks measure whether a model is correct. They do not measure whether a model is reliable. This distinction is largely academic for single-shot inference, but becomes critical for agentic AI systems, where a single rephrased prompt can trigger cascading failures in multi-step execution. Yet this form of instability is not captured by existing evaluations. Hallucinations live in variance: they arise when semantically equivalent prompts activate inconsistent internal pathways, producing divergent outputs. Consistent but incorrect outputs reflect bias or missing knowledge; confident guessing reflects calibration failure. Neither constitutes hallucination under this definition. When error is variance-dominated, reducing redundant pathways improves reliability without adding knowledge. We formalize this through Semantic Stability (SS), measured via Paraphrase Consistency (PC@k): generate k paraphrases, greedy decode each, compute mode agreement. SS is a diagnostic for variance-driven unreliability, not a method for improving correctness. We show that a dense Qwen3-0.6B agrees with itself only 23.8% of the time; at 32% sparsity, agreement jumps to 55.9%. A phase diagram reveals the sweet spot where variance reduction outpaces bias accumulation, and regimes where stability collapses onto wrong answers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。