释放多语言编码器潜力:通过概率校准提升零样本性能
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration
AI总结:
本文提出通过概率校准技术修正预训练多语言编码器在完形填空式零样本任务中对高频标签词的预测偏好,并在单语和多语编码器的零样本及少样本场景中验证了该方法能带来显著性能提升。
摘要:
预训练多语言编码器模型可以通过将输入示例重构为完形填空式提示,直接执行零样本多语言任务或语言探测。这是通过预测掩码标记位置的标签词概率来实现的,无需更新模型参数。然而,该方法的性能受限于模型对预测预训练期间频繁出现的标签词的偏好。这些词通常会获得高概率。为解决此问题,我们将模型与校准技术相结合,以修正模型预测的标签词概率。我们首先在单语编码器的零样本和少样本场景下,验证了所提出的简单校准方法及其他现有技术的有效性。随后,我们将这些校准技术应用于多语言编码器,在广泛的任务中取得了显著的性能提升。
英文摘要:
Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this method is limited by the model's bias toward predicting label words which frequently occurred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with calibration techniques which modify the probabilities of label words predicted by the models. We first validate the effectiveness of a proposed simple calibration method together with other existing techniques on monolingual encoders in both zero- and few-shot scenarios. We subsequently employ these calibration techniques on multilingual encoders, resulting in substantial performance improvements across a wide range of tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。