LLM Microscope:模型内部结构对答案正确性与上下文利用的揭示
LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
AI总结:
本研究提出LLM Microscope方法,利用大语言模型内部激活值预测输出正确性并评估上下文效用,实验验证其准确率约75%且优于提示基线,为理解LLM决策过程提供新视角。
摘要:
尽管大语言模型(LLM)具备极高的实用价值,可信度仍是核心问题:模型常以高置信度生成错误信息。虽然上下文信息有助于引导生成,但判断查询何时能从检索到的上下文获益、评估该上下文的有效性仍具挑战性。本研究将可解释性方法落地,探究能否仅通过模型激活值预测输出的正确性,同时探索模型内部是否存在反映外部上下文效用的信号。我们覆盖正确、错误与无关三类上下文,并提出区分它们的指标。在6种不同模型上的实验表明,基于首个输出token中间层激活值训练的简单分类器,能以约75%的准确率预测输出正确性,可实现早期审计。我们的基于模型内部结构的指标在区分正确与错误上下文方面显著优于提示基线,可防范受污染上下文引入的错误。这些发现为深入理解LLM的底层决策过程提供了新视角。我们的代码已在https://github.com/jiarui-liu/LLM-Microscope公开。
英文摘要:
Although large language models (LLMs) have tremendous utility, trustworthiness is still a chief concern: models often generate incorrect information with high confidence. While contextual information can help guide generation, identifying when a query would benefit from retrieved context and assessing the effectiveness of that context remains challenging. In this work, we operationalize interpretability methods to ascertain whether we can predict the correctness of model outputs from the model's activations alone. We also explore whether model internals contain signals about the efficacy of external context. We consider correct, incorrect, and irrelevant context and introduce metrics to distinguish amongst them. Experiments on six different models reveal that a simple classifier trained on intermediate layer activations of the first output token can predict output correctness with about 75% accuracy, enabling early auditing. Our model-internals-based metric significantly outperforms prompting baselines at distinguishing between correct and incorrect context, guarding against inaccuracies introduced by polluted context. These findings offer a lens to better understand the underlying decision-making processes of LLMs. Our code is publicly available at https://github.com/jiarui-liu/LLM-Microscope
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。