arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.05240cs.CL
收藏

事实探究是[ MASK ]:学习与学习回忆

Factual Probing Is [MASK]: Learning vs. Learning to Recall

Princeton University(普林斯顿大学)

Zexuan Zhong, Dan Friedman, Danqi Chen

摘要:

Petroni等人(2019)证明,可以通过将世界事实表达为完形填空风格的提示,从预先训练的语言模型中检索世界事实,并将模型的预测准确性解释为它编码的事实信息量的下限。随后的工作试图通过寻找更好的提示来收紧估计,使用不相交的事实集作为训练数据。在这项工作中,我们做了两个互补的贡献,以更好地理解这些事实探测技术。首先,我们提出了OptiPrompt,一种新的和有效的方法,直接优化连续的嵌入空间。我们发现这种简单的方法能够预测LAMA基准测试中额外的6.4%的事实。其次,我们提出了一个更重要的问题:我们真的可以把这些探测结果解释为一个下界吗?有没有可能这些搜索方法也从训练数据中学习?我们发现,有些令人惊讶的是,这些方法使用的训练数据包含一定的潜在的事实分布,和所有现有的提示方法,包括我们的,能够利用它们更好的事实预测。我们进行了一组对照实验,将“学习”与“学习回忆”分开,提供了一个更详细的图片,说明不同的提示可以揭示预训练的语言模型。

英文摘要:

Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes. Subsequent work has attempted to tighten the estimate by searching for better prompts, using a disjoint set of facts as training data. In this work, we make two complementary contributions to better understand these factual probing techniques. First, we propose OptiPrompt, a novel and efficient method which directly optimizes in continuous embedding space. We find this simple method is able to predict an additional 6.4% of facts in the LAMA benchmark. Second, we raise a more important question: Can we really interpret these probing results as a lower bound? Is it possible that these prompt-search methods learn from the training data too? We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution, and all the existing prompt methods, including ours, are able to exploit them for better fact prediction. We conduct a set of control experiments to disentangle "learning" from "learning to recall", providing a more detailed picture of what different prompts can reveal about pre-trained language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑