LLM 知识是脆弱的：真实性表征依赖于表面相似性
LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
AI总结：
本研究通过扰动测试发现，LLM 的真实性表征在输入偏离预训练分布时崩溃，表明其知识学习是浅层且依赖表面形式，导致性能脆弱，挑战了真实性探测器的可靠性。
摘要：
对于大型语言模型（LLMs）而言，若要可靠，它们必须学习能够广泛应用于多样场景（通常不同于训练时所见的场景）的稳健知识。然而，大量研究表明，LLM 的性能可能是脆弱的，模型对琐碎的输入变化表现出过度的敏感性。在这项工作中，我们探讨这种脆弱性是否是不稳定的内部知识表征的直接结果。为了探究这一问题，我们基于先前的工作，该工作表明 LLM 的表征编码了陈述的真实性——即，真实的、事实性的陈述可以容易地与虚假的、不准确的陈述区分开来。具体来说，我们通过评估在经历了表面变换（如拼写错误或改写）以使其分布外（OOD）的样本上的表征可分性，来测试所学知识的稳健性。通过应用保持语义的扰动，我们研究了在四个 LLM 家族、五个评估数据集和三种知识探测方法中，随着陈述变得更加 OOD，可分性如何退化。我们的结果显示，陈述真实性的内部表征随着样本的呈现形式与预训练期间所见的形式变得不那么相似而崩溃。虽然 LLM 在陈述与预训练数据高度相似时通常能够区分真假陈述，但这种能力高度依赖于陈述的确切表面形式。这些发现为脆弱的基准性能提供了一个可能的解释：LLM 可能学习到浅层的、非稳健的知识表征，这些表征仅允许有限的泛化能力。我们的工作对真实性探测器的实用性提出了根本性挑战，并且更广泛地呼吁进一步研究以提高所学知识表征的稳健性。
英文摘要：
For Large Language Models (LLMs) to be reliable, they must learn robust knowledge that can be generally applied in diverse settings -- often unlike those seen during training. Yet, extensive research has shown that LLM performance can be brittle, with models exhibiting excessive sensitivity to trivial input variations. In this work, we explore whether this brittleness is a direct result of unstable internal knowledge representations. To explore this question, we build on previous work showing that LLM representations encode statement truthfulness -- i.e., true, factual statements can be easily separated from false, inaccurate ones. Specifically, we test the robustness of learned knowledge by evaluating representation separability on samples that have undergone superficial transformations to drive them out-of-distribution (OOD), such as typos or reformulations. By applying semantically-preserving perturbations, we study how separability degrades as statements become more OOD, across four LLM families, five evaluation datasets, and three knowledge probing methods. Our results reveal that internal representations of statement truthfulness collapse as the samples' presentations become less similar to those seen during pre-training. While LLMs can often distinguish between true and false statements when they closely resemble the pre-training data, this ability is highly dependent on the statement's exact surface form. These findings offer a possible explanation for brittle benchmark performance: LLMs may learn shallow, non-robust knowledge representations that allow for only limited generalizability. Our work presents a fundamental challenge for the utility of truthfulness probes, and more broadly, calls for further research on improving the robustness of learned knowledge representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
