指令调优大语言模型解码时真实性方法的受控研究
A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs
AI总结：
本研究通过分析每层令牌logits特征，提出CHAIR框架检测幻觉，在TruthfulQA和MMLU上显著提升零样本检测准确率。
摘要：
在这项工作中，我们引入了CHAIR（Classifier of Hallucination As ImproveR），一个通过分析每个令牌每一层的内部logits来检测幻觉的监督框架。我们的方法从所有层的令牌logits中提取一组紧凑的特征，如最大值、最小值、均值、标准差和斜率，从而在不发生过拟合的情况下实现有效的幻觉检测。在TruthfulQA和MMLU数据集上的实验表明，CHAIR显著提高了检测准确性，特别是在零样本场景下，展示了其鲁棒性和泛化能力。除了幻觉检测，CHAIR还凸显了利用内部表示设计高级解码策略的潜力。通过利用logits中的模式，我们建议更复杂的模型和自适应解码方法可以进一步减少幻觉并提高文本完成质量。CHAIR不仅为检测幻觉提供了实用解决方案，还为探索LLM中更丰富的表示以改进其事实性和连贯性奠定了基础。
英文摘要：
Decoding-time truthfulness methods -- layer-contrast decoding, inference-time intervention, and learned logit adapters -- have demonstrated 10-30 point gains on TruthfulQA when applied to base language models. However, modern instruction-tuned LLMs already achieve substantially higher baselines (61-76%), raising the question of whether these methods remain effective in practice. We design a six-control evaluation framework -- out-of-distribution training, multi-judge validation, simple decoding baselines, confound controls, bootstrap confidence intervals, and seed variance -- and apply it across 5 models (1B-70B), 3 benchmarks, and 15 methods. We find that previously reported gains shrink substantially under strict controls: on the full TruthfulQA benchmark (N=817), no token-level method achieves statistically significant improvement, and the best learned adapter scores -2.0 points below greedy (p=.23). We identify five evaluation sensitivities -- contamination, judge choice, missing baselines, confounds, and statistical noise -- that individually or jointly account for these discrepancies. Cross-benchmark validation on HaluEval QA and TriviaQA confirms that these patterns extend beyond TruthfulQA. Deliberative prompting methods (chain-of-thought, self-critique) appear more robust in the evaluated regime, with CoT achieving +5.6-19pp across benchmarks as a training-free, single-pass method. We release a seven-point evaluation checklist and discuss implications for future truthfulness research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
