KERMIT:基于生成插入的序列建模
KERMIT: Generative Insertion-Based Modeling for Sequences
摘要:
我们提出了KERMIT,一个简单的基于插入的方法生成建模的序列和序列对。KERMIT对联合分布及其分解(即,边缘和条件)使用单个神经网络,并且与许多先前的工作不同,不依赖于数据分布的预先指定的因子分解。在训练期间,可以馈送KERMIT配对数据$(x, y)$以学习联合分布$p(x, y)$,并且可选地混入未配对数据$x$或$y$以细化边缘值$p(x)$或$p(y)$。在推理过程中,我们可以从两个方向访问条件句$p(x \mid y)$和$p(y \mid x)$。我们也可以从联合分布或边缘分布中取样。该模型支持串行完全自回归解码和并行部分自回归解码,后者表现出经验对数运行时间。我们通过机器翻译,表征学习,和zero-shot完形填空问题的回答,我们的统一的方法是能够匹配或超过性能的专用国家的最先进的系统在广泛的任务,而不需要特定于问题的架构适应的实验证明。
英文摘要:
We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。