利用非纠缠表示提高NLP模型的鲁棒性和通用性
Improving Robustness and Generality of NLP Models Using Disentangled Representations
摘要：
监督神经网络首先将输入$x$映射到单个表示$z$，然后将$z$映射到输出标签$y$，在广泛的自然语言处理（NLP）任务中取得了显著的成功。尽管取得了成功，但神经模型缺乏鲁棒性和通用性：对输入的微小扰动可能会导致绝对不同的输出;在一个域上训练的模型在另一个域上测试时，性能急剧下降。在本文中，我们提出的方法，以提高鲁棒性和通用性的NLP模型的立场，解开表示学习。所提出的策略不是将$x$映射到单个表示$z$，而是将$x$映射到一组表示$\{z_1,z_2,...,z_K\}$，同时强制它们被解开。然后将这些表示映射到不同的logit $l$，其集合用于进行最终预测$y$。我们提出了不同的方法将这一想法纳入目前广泛使用的模型，包括添加一个$L$ 2正则化的$z$ s或添加总相关（TC）的变分信息瓶颈（VIB）的框架下。我们表明，该标准训练的模型提供了更好的鲁棒性和域适应能力，在广泛的监督学习任务。
英文摘要：
Supervised neural networks, which first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$, have achieved remarkable success in a wide range of natural language processing (NLP) tasks. Despite their success, neural models lack for both robustness and generality: small perturbations to inputs can result in absolutely different outputs; the performance of a model trained on one domain drops drastically when tested on another domain. In this paper, we present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. Instead of mapping $x$ to a single representation $z$, the proposed strategy maps $x$ to a set of representations $\{z_1,z_2,...,z_K\}$ while forcing them to be disentangled. These representations are then mapped to different logits $l$s, the ensemble of which is used to make the final prediction $y$. We propose different methods to incorporate this idea into currently widely-used models, including adding an $L$2 regularizer on $z$s or adding Total Correlation (TC) under the framework of variational information bottleneck (VIB). We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
