发表机构
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过两阶段训练模型，发现事实的检索能力取决于训练时的请求形式，并受上下文状态影响，可跨事实迁移。
AI 中文摘要
一个在“X的首都是Y”上训练的模型，可能在“X的首都是”之后生成“Y”，但在“X的首都：”之后却无法生成。我们将这些引出同一事实的不同方式称为请求形式。为了将学习事实与检索事实分开，我们分两个阶段训练了两个模型。在第一阶段（请求形式训练），一个模型以五种形式看到每个事实，而另一个模型仅以陈述句形式看到相同的事实。在第二阶段（目标事实训练），两者都接受关于新事实的相同训练，且全部以陈述句形式呈现。随后，两者从陈述句中检索新事实的能力几乎相同，但在其他请求形式上差异显著。因此，模型可以在学习事实之前，通过一种请求形式学会如何检索。为了理解这种差异，我们检查了答案之前紧邻的隐藏状态，称之为上下文状态。当给定同一事实的两种不同请求形式时，在第一阶段接受五种形式训练的模型，比在该阶段仅接受陈述句训练的模型，产生更相似的上下文状态。在检索时改变该状态，可以启用或阻止对已学习事实的检索，且这种效果可跨事实和事实关系（如首都和货币）迁移。为了测试其在学习中的作用，我们仅在目标事实训练期间改变上下文状态。这种干预改变了后续的检索，而测试时无需干预。综合这些结果表明，后续检索依赖于早期的请求形式经验以及事实学习期间的上下文状态。
英文摘要
A model trained on "The capital of X is Y" may produce "Y" after "The capital of X is" but fail after "The capital of X:". We call these different ways of eliciting the same fact request forms. To separate learning a fact from retrieving it, we train two models in two stages. In the first stage (request-form training), one model sees each fact in five forms and the other sees the same facts only as statements. In the second stage (target-fact training), both receive identical training on new facts, all as statements. Both then retrieve the new facts almost equally well from statements, but differ sharply on other request forms. Thus, a model can learn how to retrieve through a request form before it learns the facts. To understand this difference, we examine the hidden state immediately before the answer, which we call the context state. When given two different request forms for the same fact, the model trained on five forms in stage one produces more similar context states than the model trained on statements alone in that stage. Changing this state at retrieval time can enable or prevent retrieval of an already learned fact, and the same effect transfers across facts and factual relations, such as capitals and currencies. To test its role during learning, we change the context state only during target-fact training. This intervention changes later retrieval without intervention at test time. Together, these results show that later retrieval depends on earlier request-form experience and the context state during fact learning.
