超越统计显著性：量化多语言和多任务NLP评估中的不确定性与统计变异
Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluation
AI总结：
本文提出基于重采样的方法，用于量化多语言和多任务NLP评估中的不确定性与统计变异，以提高评估结果的可靠性。
摘要：
我们介绍了一套基于重采样的方法，用于量化多语言和/或多任务NLP基准中评估指标的不确定性和统计精度。我们展示了性能评分中的实验变异源于模型和数据相关的原因，并表明考虑两者是避免在假设重复实验中大幅低估总体变异的必要条件。通过多语言问答、机器翻译和命名实体识别等示例任务，我们还展示了重采样方法在量化领导者板上各种量（如模型排名和模型间的成对差异）的重复不确定性方面的有效性。
英文摘要：
We introduce a set of resampling-based methods for quantifying uncertainty and statistical precision of evaluation metrics in multilingual and/or multitask NLP benchmarks. We show how experimental variation in performance scores arises from both model and data-related sources, and that accounting for both of them is necessary to avoid substantially underestimating the overall variability over hypothetical replications. Using multilingual question answering, machine translation, and named entity recognition as example tasks, we also demonstrate how resampling methods are useful for quantifying the replication uncertainty of various quantities used in leaderboards such as model rankings and pairwise differences between models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
