神经多样性调节语言模型的幻觉
Neural Diversity Regularizes Hallucinations in Language Models
AI总结：
神经多样性通过解相关并行表示减少语言模型幻觉，结合LoRA适配器与Barlow Twins正则化，提升可靠性并保持准确性。
摘要：
尽管参数、计算和数据量不断增加，语言模型仍然会产生幻觉。我们提出神经多样性——解相关并行表示——作为一种有原则的机制，可以在固定参数和数据预算下减少幻觉率。虽然现有的缓解策略大多针对准确性，我们为集合语言模型的幻觉概率提供了首次正式的尾部界限，将其重新表述为一个二阶矩可靠性问题，并解释了在并行配置中观察到的94.3%的实证可靠性变化。我们引入了ND-LoRA（神经多样性低秩适应），结合并行LoRA适配器与Barlow Twins正则化，同时在保持一般准确性的情况下，将幻觉减少了最高25.6%（平均减少14.6%）。消融研究显示LoRA适配器和正则化作用是协同的，因果干预证明神经多样性是中介因素，相关性研究表明规模：神经相关性增加0.1%与幻觉增加3.8%相关。最后，任务依赖的最优性显现：不同任务需要不同的最优神经多样性量。总的来说，我们的结果强调神经多样性作为第三轴的扩展——与参数和数据正交——以在固定预算下提高语言模型的可靠性。
英文摘要：
Language models continue to hallucinate despite increases in parameters, compute, and data. We propose neural diversity -- decorrelated parallel representations -- as a principled mechanism that reduces hallucination rates at fixed parameter and data budgets. While existing mitigation strategies largely target accuracy, we provide the first formal tail bounds for hallucination probability in ensembled language models, reframing it as a second-moment reliability problem and explaining 94.3% of empirical reliability variation seen across parallel configurations. We introduce ND-LoRA (Neural Diversity Low-Rank Adaptation), combining parallel LoRA adapters with Barlow Twins regularization, and reduce hallucinations by up to 25.6% (and 14.6% on average) while preserving general accuracy. Ablations show LoRA adapters and regularization act synergistically, causal interventions prove neurodiversity as the mediating factor and correlational studies indicate scale: a 0.1% neural correlation increase is associated with a 3.8% hallucination increase. Finally, task-dependent optimality emerges: different tasks require different optimal amounts of neurodiversity. Together, our results highlight neural diversity as a third axis of scaling -- orthogonal to parameters and data -- to improve the reliability of language models at fixed budgets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
