arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.05159cs.CLcs.AI
收藏

语言模型能否做到具体明确?如何实现?

Can Language Models Be Specific? How?

University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University at Buffalo(布法罗大学);NVIDIA(英伟达)

Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, Wen-mei Hwu

AI总结:

本文提出通过掩码预测基准衡量预训练语言模型的具体性,发现现有模型仅轻微偏好具体答案,并设计两种无需训练的提示方法有效提升模型语言的具体性。

摘要:

“他是一个人”,“巴黎位于地球上”。这两句话都正确但毫无意义——因为缺乏具体性。在本文中,我们提出衡量预训练语言模型(PLMs)语言具体性的方法。为此,我们引入一种新方法,通过构建带提示的掩码标记预测任务来建立具体性测试基准。例如,给定“多伦多位于 [MASK]。”,我们想测试 PLMs 是否更倾向于填入更具体的答案,例如安大略省而不是加拿大。根据我们的评估,现有 PLMs 对更具体答案只有轻微的偏好。我们识别了影响具体性的潜在因素,并设计了两种基于提示的方法来提高具体性。结果表明,所提出的方法无需额外训练即可提高模型的具体性。我们希望这项工作能引起人们对语言模型具体性概念的关注,并鼓励研究界进一步探索这一重要但尚未被充分研究的问题。

英文摘要:

"He is a person", "Paris is located on the earth". Both statements are correct but meaningless - due to lack of specificity. In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is. To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. For instance, given "Toronto is located in [MASK].", we want to test whether a more specific answer will be better filled in by PLMs, e.g., Ontario instead of Canada. From our evaluations, we show that existing PLMs have only a slight preference for more specific answers. We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity. Results show that the specificity of the models can be improved by the proposed methods without additional training. We hope this work can bring to awareness the notion of specificity of language models and encourage the research community to further explore this important but understudied problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑