How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a target benchmark. Likelihood training shapes the model's predictive distribution from the pre-training corpus, and zero-shot scores under likelihood-based evaluation depend in part on the lexical alignment between that corpus and the benchmark target. Cleanly tracing how the lexical imprint of pre-training data scales with corpus--benchmark alignment requires controlled from-scratch pre-training that varies corpus identity while holding the training recipe fixed. Across this controlled sweep, spanning $10$ zero-shot benchmarks, $4$ pre-training corpora, and $3$ model scales, we find that corpus--benchmark lexical alignment, UCE, consistently tracks zero-shot accuracy on every benchmark, with corpora more aligned with the benchmark (lower UCE) achieving higher accuracy. The same signal extends from measurement to control at finer scopes. At the document level, strengthening the imprint by selecting training documents to maximize lexical alignment with the benchmark produces the accuracy gains predicted by the mechanism. At the prompt level, matching the imprint per prompt by routing each prompt to the drafter whose domain corpus is most lexically aligned with it produces measurable throughput gains over a generalist-drafter baseline. Taken together, pre-training corpora leave a measurable lexical imprint on base-model zero-shot behavior, and UCE provides a model-free probe of lexical alignment between reference and target data.
ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5
Charles University（查理大学）
我们在W-NUT 2021上展示了多语言词汇规范化（MultiLexNorm）共享任务的获奖作品（van der Goot等人，2021 a），它评估了11种语言的12个社交媒体数据集的词汇规范化系统。我们的解决方案基于预先训练的字节级语言模型ByT 5（Xue et al.，2021 a），我们进一步对合成数据进行预训练，然后对真实的归一化数据进行微调。我们的系统实现了最好的性能，通过一个很大的利润率在内在的评价，也是最好的性能，通过依存句法分析的外在评价。源代码发布于https://github.com/ufal/multilexnorm2021，微调模型发布于https://huggingface.co/ufal。
Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models
Boston University（波士顿大学）；New York University（纽约大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Microsoft Research（微软研究院）
人类的语言能力通常以组合性和它所能实现的概括为特征--人类学习者可以通过组合已知的部分来产生和理解新颖的复杂表达。几个基准测试在训练和测试中利用分布式控制来衡量合成泛化，其中某些词汇项仅在训练期间出现在有限的上下文中。虽然最近使用这些基准测试的工作表明，预训练模型实现了令人印象深刻的泛化性能，但我们认为，暴露于预训练数据可能会打破上述分布控制。使用Kim和Linzen（2020）的COGS基准，我们测试了两种改进的评估设置来控制这个问题：（1）用新的字符序列替换上下文控制的词汇项，以及（2）用由新的嵌入表示的特殊标记替换它们。我们发现，这两种设置都导致T5的泛化性能较低（Raffel et al.，2020年），这表明先前报道的结果被高估，由于不受控制的词汇暴露在预训练。对于新的嵌入，性能下降更极端，并且随着预训练数据的量而增加，突出了逆缩放的有趣情况。
