论 NLP 中的“科学债务”：在语言模型预训练研究中倡导更严谨的理由
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research
AI总结：
本文批评 PLM 预训练研究缺乏消融与公平比较，并通过 BERT、ELMo 和 GPT-1 的对照实验表明严谨基线可取得相当或更优结果，进而提出提升研究系统性的建议。
摘要：
这篇基于证据的立场论文批评了语言模型预训练文献中的当前研究实践。尽管日益优秀的预训练语言模型（PLM）带来了近期的快速进展，但当前的 PLM 研究实践常常混淆模型改进的不同可能来源，既没有开展恰当的消融研究，也没有在可比条件下对不同模型进行有原则的比较。这些实践：（i）使我们难以理解应在何种情形下使用哪些预训练方法；（ii）阻碍可复现性与功劳归属；（iii）令理解“每个因素究竟如何促成我们今天所取得的进展？”变得困难。我们通过重新审视 BERT 相对于其基线 ELMo 和 GPT-1 的成功提供了一个典型例证，并证明在基线被调优到相近程度的可比条件下，这些基线（乃至其更简单的变体）实际上能够取得与 BERT 相当甚至更好的性能。这些发现表明，厘清模型改进的不同因素如何能够带来有价值的新洞见。最后，我们就如何鼓励和激励这一研究方向提出建议，以加速进展，更系统、更深入地理解当今哪些因素推动了基础模型的进步。
英文摘要：
This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: "How exactly does each factor contribute to the progress that we have today?" We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how -- under comparable conditions where the baselines are tuned to a similar extent -- these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
