Metadata Shaping:尾部的自然语言注释
Metadata Shaping: Natural Language Annotations for the Tail
摘要:
语言模型(LM)已经取得了显著的进步,但仍然难以推广到训练数据之外的罕见语言模式。由于罕见的实体和事实是普遍的查询用户提交到流行的应用程序,如搜索和个人助理系统,提高LMs可靠地捕获知识的能力,在罕见的实体是一个紧迫的挑战,在重要的先前工作研究。注意到现有的方法主要是修改LM架构或引入辅助目标来注入有用的实体知识,我们问到什么程度,我们可以匹配这些架构的质量使用一个基础LM架构,只改变数据?我们提出了元数据整形,一种方法,其中容易获得的元数据,如实体描述和分类标签,被附加到基于信息理论度量的例子。直观地,如果对应于流行实体的元数据与稀有实体的元数据重叠,则LM可以能够使用从类似流行实体学习的模式来更好地推理稀有实体。在标准的实体丰富的任务(TACRED,FewRel,OpenEntity)中,在没有对LM进行任何更改的情况下,元数据整形超过BERT基线高达5.3个F1点,并实现或与最先进的结果竞争。我们进一步表明,在包含tail与流行实体的示例中,改进高达10倍。
英文摘要:
Language models (LMs) have made remarkable progress, but still struggle to generalize beyond the training data to rare linguistic patterns. Since rare entities and facts are prevalent in the queries users submit to popular applications such as search and personal assistant systems, improving the ability of LMs to reliably capture knowledge over rare entities is a pressing challenge studied in significant prior work. Noticing that existing approaches primarily modify the LM architecture or introduce auxiliary objectives to inject useful entity knowledge, we ask to what extent we could match the quality of these architectures using a base LM architecture, and only changing the data? We propose metadata shaping, a method in which readily available metadata, such as entity descriptions and categorical tags, are appended to examples based on information theoretic metrics. Intuitively, if metadata corresponding to popular entities overlap with metadata for rare entities, the LM may be able to better reason about the rare entities using patterns learned from similar popular entities. On standard entity-rich tasks (TACRED, FewRel, OpenEntity), with no changes to the LM whatsoever, metadata shaping exceeds the BERT-baseline by up to 5.3 F1 points, and achieves or competes with state-of-the-art results. We further show the improvements are up to 10x larger on examples containing tail versus popular entities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。