大语言模型的阅读理解受参数化知识影响，并在假设性陈述上表现困难
LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements
AI总结：
本文提出在虚构数据上进行阅读理解以消除参数化知识干扰，测试ChatGPT、GPT-4、LLaMA 2和Mixtral后发现LLMs在模态和条件等假设性语境中表现显著下降，并再次暴露知识冲突脆弱性。
摘要：
阅读理解（RC）任务通常以基于上下文的问答（QA）形式实现，是评估语言模型自然语言理解（NLU）能力的主要手段。然而，当应用于具备广泛内置世界知识的大语言模型（LLMs）时，这种方法可能具有欺骗性。如果上下文与LLMs的内部知识一致，则难以辨别模型的答案究竟源于对上下文的理解，还是源于LLMs的内部信息。相反，使用与模型知识相冲突的数据会产生错误趋势，从而扭曲结果。为解决这一问题，我们建议在虚构数据上进行阅读理解，这些数据基于虚构的事实和实体。该任务完全独立于模型的世界知识，使我们能够在不受参数化知识干扰的情况下评估LLMs的语言能力。我们在此类虚构数据上测试了ChatGPT、GPT-4、LLaMA 2和Mixtral，发现了一类对当前LLMs构成挑战的语言现象，涉及以替代性、假设性场景进行思考。虽然所有模型在处理简单的肯定和否定语境时都表现出较高的准确率，但在处理模态和条件语境时则更容易出错。关键的是，这些现象也再次触发了LLMs对知识冲突的脆弱性。特别是，虽然一些模型在肯定和否定语境中几乎不受知识冲突的影响，但当面对语义上更复杂的模态和条件环境时，它们往往无法将文本与自身内部知识区分开来。
英文摘要：
The task of reading comprehension (RC), often implemented as context-based question answering (QA), provides a primary means to assess language models' natural language understanding (NLU) capabilities. Yet, when applied to large language models (LLMs) with extensive built-in world knowledge, this method can be deceptive. If the context aligns with the LLMs' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from LLMs' internal information. Conversely, using data that conflicts with the models' knowledge creates erroneous trends which distort the results. To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities. This task is entirely independent of the models' world knowledge, enabling us to evaluate LLMs' linguistic abilities without the interference of parametric knowledge. Testing ChatGPT, GPT-4, LLaMA 2 and Mixtral on such imaginary data, we uncover a class of linguistic phenomena posing a challenge to current LLMs, involving thinking in terms of alternative, hypothetical scenarios. While all the models handle simple affirmative and negative contexts with high accuracy, they are much more prone to error when dealing with modal and conditional contexts. Crucially, these phenomena also trigger the LLMs' vulnerability to knowledge-conflicts again. In particular, while some models prove virtually unaffected by knowledge conflicts in affirmative and negative contexts, when faced with more semantically involved modal and conditional environments, they often fail to separate the text from their internal knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
