有效的语言模型预训练和下游适应技巧包：GLUE案例研究
Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
摘要：
这份技术报告简要介绍了我们的JDExplore d-team在通用语言理解评估（GLUE）排行榜上提交的Vega v1，其中GLUE是九个自然语言理解任务的集合，包括问答、语言可接受性、情感分析、文本相似性、释义检测和自然语言推理。[方法]研究几种有效的策略，并选择其最佳组合设置作为训练配方。在模型结构上，我们采用了具有分离注意力的普通Transformer作为基本的块编码器。对于自监督训练，我们采用代表性的去噪目标（即，替换令牌检测）并将对比目标（即，句子嵌入对比学习）与其在阶段2中。在微调过程中，采用了一些先进的技术，如转导微调，自校准微调，对抗微调。[结果]根据我们的提交记录（2022年1月），通过我们优化的预训练和微调策略，我们的13亿模型在4/9的任务上设置了新的最先进水平，达到了91.3的最佳平均得分。令人鼓舞的是，我们的Vega v1是第一个在两项具有挑战性的任务上超越人类的强大性能，即，SST-2和WNLI。我们相信我们经验上成功的配方和一袋技巧可以为开发有效的区分大语言模型提供新的思路。
英文摘要：
This technical report briefly describes our JDExplore d-team's submission Vega v1 on the General Language Understanding Evaluation (GLUE) leaderboard, where GLUE is a collection of nine natural language understanding tasks, including question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference. [Method] We investigate several effective strategies and choose their best combination setting as the training recipes. As for model structure, we employ the vanilla Transformer with disentangled attention as the basic block encoder. For self-supervised training, we employ the representative denoising objective (i.e., replaced token detection) in phase 1 and combine the contrastive objective (i.e., sentence embedding contrastive learning) with it in phase 2. During fine-tuning, several advanced techniques such as transductive fine-tuning, self-calibrated fine-tuning, and adversarial fine-tuning are adopted. [Results] According to our submission record (Jan. 2022), with our optimized pretraining and fine-tuning strategies, our 1.3 billion model sets new state-of-the-art on 4/9 tasks, achieving the best average score of 91.3. Encouragingly, our Vega v1 is the first to exceed powerful human performance on the two challenging tasks, i.e., SST-2 and WNLI. We believe our empirically successful recipe with a bag of tricks could shed new light on developing efficient discriminative large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
