通过真实性增强预训练语言模型中的事实检索
Enhancing Fact Retrieval in PLMs through Truthfulness
AI总结：
本文提出利用辅助模型基于PLMs隐藏状态评估输入真实性，从而将事实检索性能提升高达33%。
摘要：
预训练语言模型（PLMs）在预训练阶段通过预测句子中的下一个词或缺失词，编码了关于世界的各种事实。由于PLMs被设想为可以用自然语言查询的软知识库，人们一直关注量化和提高从PLMs中可提取的事实数量。目前存在不同的方法来增强从PLM中进行事实检索。最近的研究表明，可以利用PLMs的隐藏状态来判断PLMs输入的真实性。然而，利用这一发现来改进事实知识检索的工作尚未被探索。在这项工作中，我们研究了使用一个辅助模型来改进事实检索。该辅助模型基于PLMs对应的隐藏状态表示来评估输入的真实性。我们在多个掩码语言模型（masked PLMs）上评估了该方法，并表明它可以将事实检索性能提升高达33%。我们的发现凸显了PLMs的隐藏状态表示在改进其事实知识检索方面的潜力。
英文摘要：
Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
