arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.02870cs.CLcs.SE
收藏

非参数语言模型中结构局部性的捕获

Capturing Structural Locality in Non-parametric Language Models

School of Computer Science(计算机科学学院);Carnegie Mellon University(卡内基梅隆大学)

Frank F. Xu, Junxian He, Graham Neubig, Vincent J. Hellendoorn

摘要:

结构局部性是真实世界数据集的普遍特征,其中数据点被组织成局部层次结构。一些例子包括文本中的主题集群或源代码存储库中的项目层次结构。在本文中,我们探讨利用这种结构的局部性内的非参数语言模型,生成序列,引用检索到的例子从外部源。我们提出了一种简单而有效的方法,通过添加学习参数,提高从本地邻域检索示例的可能性,将局部信息添加到这样的模型中。在两个不同的领域,Java源代码和维基百科的文本,实验表明,局部性功能提高模型的效率模型没有访问这些功能,有趣的差异。我们还进行了分析,如何以及在何处局部性功能有助于提高性能,以及为什么传统上使用的上下文相似性度量不足以掌握的局部性结构。

英文摘要:

Structural locality is a ubiquitous feature of real-world datasets, wherein data points are organized into local hierarchies. Some examples include topical clusters in text or project hierarchies in source code repositories. In this paper, we explore utilizing this structural locality within non-parametric language models, which generate sequences that reference retrieved examples from an external source. We propose a simple yet effective approach for adding locality information into such models by adding learned parameters that improve the likelihood of retrieving examples from local neighborhoods. Experiments on two different domains, Java source code and Wikipedia text, demonstrate that locality features improve model efficacy over models without access to these features, with interesting differences. We also perform an analysis of how and where locality features contribute to improved performance and why the traditionally used contextual similarity metrics alone are not enough to grasp the locality structure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑