神经序列模型的计算与质量控制
Controlling Computation versus Quality for Neural Sequence Models
摘要：
大多数神经网络对每个例子都使用相同的计算量，而与输入的固有复杂性无关。此外，使计算量适应示例的方法集中于每个示例找到固定的推理时间计算图，忽略任何外部计算预算或变化的推理时间限制。在这项工作中，我们利用条件计算，使神经序列模型（Transformer）更有效和计算感知推理。我们首先修改了Transformer架构，使每组操作都可以根据学习的控制网络的输出有条件地执行。然后，我们在多任务设置中训练该模型，其中每个任务对应于特定的计算预算。这允许我们训练单个模型，该模型可以被控制以在计算质量权衡曲线的不同点上操作，这取决于推断时的可用计算预算。我们在两个任务上评估我们的方法：（i）WMT英法翻译和（ii）无监督表示学习（BERT）。我们的实验表明，所提出的条件计算Transformer（CCT）是有竞争力的香草Transformers时，允许利用其全部计算预算，同时提高显着的计算等效基线时，较小的计算预算。
英文摘要：
Most neural networks utilize the same amount of compute for every example independent of the inherent complexity of the input. Further, methods that adapt the amount of computation to the example focus on finding a fixed inference-time computational graph per example, ignoring any external computational budgets or varying inference time limitations. In this work, we utilize conditional computation to make neural sequence models (Transformer) more efficient and computation-aware during inference. We first modify the Transformer architecture, making each set of operations conditionally executable depending on the output of a learned control network. We then train this model in a multi-task setting, where each task corresponds to a particular computation budget. This allows us to train a single model that can be controlled to operate on different points of the computation-quality trade-off curve, depending on the available computation budget at inference time. We evaluate our approach on two tasks: (i) WMT English-French Translation and (ii) Unsupervised representation learning (BERT). Our experiments demonstrate that the proposed Conditional Computation Transformer (CCT) is competitive with vanilla Transformers when allowed to utilize its full computational budget, while improving significantly over computationally equivalent baselines when operating on smaller computational budgets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
