TangoBERT：使用级联架构降低推理成本
TangoBERT: Reducing Inference Cost by using Cascaded Architecture
摘要：
大型基于变压器的模型，如BERT，RoBERTa和XLNet在许多NLP任务中取得了显着的成功，由于其高计算负载和能耗，货币和环境成本大幅增加。为了减少推理时间的计算负荷，我们提出了TangoBERT，级联模型架构，其中实例首先由高效但不太准确的第一层模型处理，并且只有部分实例由效率较低但更准确的第二层模型额外处理。是否应用第二层模型的决定基于由第一层模型产生的置信度分数。与基于多层Transformer模型的标准级联方法相比，我们的简单方法具有几个吸引人的实际优势。首先，它能够实现更高的加速增益（平均更低的延迟）。其次，它利用批量优化级联，这增加了相对的推理成本降低。我们报告了TangoBERT推理CPU在四个文本分类GLUE任务和一个阅读理解任务上的加速。实验结果表明，TangoBERT的性能优于高效的早期退出基线模型;在SST-2任务上，它实现了93.9%的准确率，CPU加速比为8.2倍。
英文摘要：
The remarkable success of large transformer-based models such as BERT, RoBERTa and XLNet in many NLP tasks comes with a large increase in monetary and environmental cost due to their high computational load and energy consumption. In order to reduce this computational load in inference time, we present TangoBERT, a cascaded model architecture in which instances are first processed by an efficient but less accurate first tier model, and only part of those instances are additionally processed by a less efficient but more accurate second tier model. The decision of whether to apply the second tier model is based on a confidence score produced by the first tier model. Our simple method has several appealing practical advantages compared to standard cascading approaches based on multi-layered transformer models. First, it enables higher speedup gains (average lower latency). Second, it takes advantage of batch size optimization for cascading, which increases the relative inference cost reductions. We report TangoBERT inference CPU speedup on four text classification GLUE tasks and on one reading comprehension task. Experimental results show that TangoBERT outperforms efficient early exit baseline models; on the the SST-2 task, it achieves an accuracy of 93.9% with a CPU speedup of 8.2x.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
