基于Dirichlet先验RNN的SLU中未知概念学习的令牌级不确定性建模
Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN
摘要:
现代个人助理中的口语语言理解(SLU)的一个主要任务是从话语中提取语义概念,称为槽填充。虽然现有的槽填充模型试图改善提取新的概念,没有看到在训练数据,在实践中的性能仍然不能令人满意。最近的研究收集了问题和答案注释的数据,以了解什么是未知的,应该问,但由于沉重的数据汇集工作,实际上是不可扩展的。在本文中,我们将softmax为基础的插槽填充神经架构,以模拟序列的不确定性,没有问题的监督。我们设计了一个Dirichlet先验RNN模型,通过退化作为RNN模型训练的softmax层来建模高阶不确定性。为了进一步增强不确定性建模的鲁棒性,我们提出了一种新的多任务训练来校准狄利克雷浓度参数。我们收集看不见的概念,从SLU基准数据集Snips和ATIS创建两个测试数据集。在这两个和另一个现有的概念学习基准数据集上,我们表明,我们的方法比最先进的方法显著高出8.18%。我们的方法是通用的,可以应用于任何RNN或Transformer的插槽填充模型与softmax层。
英文摘要:
One major task of spoken language understanding (SLU) in modern personal assistants is to extract semantic concepts from an utterance, called slot filling. Although existing slot filling models attempted to improve extracting new concepts that are not seen in training data, the performance in practice is still not satisfied. Recent research collected question and answer annotated data to learn what is unknown and should be asked, yet not practically scalable due to the heavy data collection effort. In this paper, we incorporate softmax-based slot filling neural architectures to model the sequence uncertainty without question supervision. We design a Dirichlet Prior RNN to model high-order uncertainty by degenerating as softmax layer for RNN model training. To further enhance the uncertainty modeling robustness, we propose a novel multi-task training to calibrate the Dirichlet concentration parameters. We collect unseen concepts to create two test datasets from SLU benchmark datasets Snips and ATIS. On these two and another existing Concept Learning benchmark datasets, we show that our approach significantly outperforms state-of-the-art approaches by up to 8.18%. Our method is generic and can be applied to any RNN or Transformer based slot filling models with a softmax layer.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。