reStructured预培训
reStructured Pre-training
摘要:
在这项工作中,我们试图破译过去几十年来NLP技术发展的内在联系,寻找本质,这为我们提供了一种(潜在的)新的NLP任务学习范式,称为reStructured Pre-training(reStructured Pre-training,简称RESTRUCTURE)。在这种范式中,数据的作用将被重新强调,模型的预训练和下游任务的微调被视为数据存储和访问的过程。在此基础上,我们操作的简单原则,一个好的存储机制不仅要有能力缓存大量的数据,但也考虑到访问的方便性。我们通过在重组数据上预训练模型来实现这一目标,重组数据由各种有价值的信息组成,而不是在克服了几个工程挑战后的原始数据。从实验上看,ESTA模型不仅超过了强大的竞争对手(例如,T0)在52/55的流行数据集上从各种NLP任务中,而且在中国最权威的考试全国高考-英语(高考-英语)中也取得了优异的成绩。具体而言,所提出的系统秦实现40分以上的平均成绩由学生和15分以上的GPT 3与1/16的参数。特别是2018年的英语考试(国家试卷Ⅲ),秦同学拿到了138.5分的高分(满分150分)。我们发布了高考基准测试,并提供了在线提交平台。此外,我们在几天前(2022.06.08)发生的2022年高考英语中测试了我们的模型,它得到了134分的总分(v.s. GPT 3的108)。
英文摘要:
In this work, we try to decipher the internal connection of NLP technology development in the past decades, searching for essence, which rewards us with a (potential) new learning paradigm for NLP tasks, dubbed as reStructured Pre-training (RST). In such a paradigm, the role of data will be re-emphasized, and model pre-training and fine-tuning of downstream tasks are viewed as a process of data storing and accessing. Based on that, we operationalize the simple principle that a good storage mechanism should not only have the ability to cache a large amount of data but also consider the ease of access. We achieve this by pre-training models over restructured data that consist of a variety of valuable information instead of raw data after overcoming several engineering challenges. Experimentally, RST models not only surpass strong competitors (e.g., T0) on 52/55 popular datasets from a variety of NLP tasks, but also achieve superior performance in National College Entrance Examination - English (Gaokao-English),the most authoritative examination in China. Specifically, the proposed system Qin achieves 40 points higher than the average scores made by students and 15 points higher than GPT3 with 1/16 parameters. In particular, Qin gets a high score of 138.5 (the full mark is 150) in the 2018 English exam (national paper III). We have released the Gaokao Benchmark with an online submission platform. In addition, we test our model in the 2022 College Entrance Examination English that happened a few days ago (2022.06.08), and it gets a total score of 134 (v.s. GPT3's 108).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。