arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.11453cs.CL
收藏

跨任务和语言的零样本学习的参数空间分解

Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);Technion, IIT(以色列理工学院)

Edoardo M. Ponti, Ivan Vulić, Ryan Cotterell, Marinela Parovic, Roi Reichart, Anna Korhonen

摘要:

大多数NLP任务和语言种类的组合都缺乏用于监督训练的域内示例,因为缺乏注释数据。神经模型如何从具有可用数据的任务语言组合到低资源组合进行样本有效的概括?在这项工作中,我们提出了一个贝叶斯生成模型的神经参数的空间。我们假设这个空间可以分解为每种语言和每项任务的潜在变量。我们通过变分推理,根据所看到的任务语言组合的数据来推断这些潜变量的后验。这使得能够在预测时间对看不见的组合进行zero-shot分类。例如,给定越南语中命名实体识别(NER)和Wolof中词性(POS)标记的训练数据,我们的模型可以对Wolof中的NER进行准确的预测。特别是,我们的实验与33种语言,从4大洲和11个家庭的类型多样的样本,并表明我们的模型产生可比或更好的结果比国家的最先进的,zero-shot跨语言迁移方法。此外,我们证明了近似贝叶斯模型平均结果更平滑的预测分布,其熵与准确性呈负相关。因此,所提出的框架还提供了预测不确定性的鲁棒估计。我们的代码位于github.com/cambridgeltl/parameter-factorization

英文摘要:

Most combinations of NLP tasks and language varieties lack in-domain examples for supervised training because of the paucity of annotated data. How can neural models make sample-efficient generalizations from task-language combinations with available data to low-resource ones? In this work, we propose a Bayesian generative model for the space of neural parameters. We assume that this space can be factorized into latent variables for each language and each task. We infer the posteriors over such latent variables based on data from seen task-language combinations through variational inference. This enables zero-shot classification on unseen combinations at prediction time. For instance, given training data for named entity recognition (NER) in Vietnamese and for part-of-speech (POS) tagging in Wolof, our model can perform accurate predictions for NER in Wolof. In particular, we experiment with a typologically diverse sample of 33 languages from 4 continents and 11 families, and show that our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods. Moreover, we demonstrate that approximate Bayesian model averaging results in smoother predictive distributions, whose entropy inversely correlates with accuracy. Hence, the proposed framework also offers robust estimates of prediction uncertainty. Our code is located at github.com/cambridgeltl/parameter-factorization

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑