用于透明比较多语言 NLP 数据集语言多样性的一种度量
A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets
AI总结：
本文提出基于参考语言样本、Jaccard 指数与文本特征的语言多样性度量，用于解释并比较多语言 NLP 数据集，分析九个常用数据集并发现多数缺少（多式）综合语。
摘要：
类型学上多样化的基准正越来越多地被创建，以追踪多语言 NLP 所取得的进展。这些数据集的语言多样性通常以样本中包含的语言或语系数量来衡量，但此类度量没有考虑所包含语言的结构属性。在本文中，我们提出对照一个参考语言样本来评估数据集的语言多样性，将其作为长期最大化语言多样性的一种手段。我们将语言表示为特征集合，并采用适用于比较度量集合的 Jaccard 指数版本。除了从类型学数据库中提取的特征之外，我们还提出了一种基于文本的自动度量，可将其用作克服人工收集特征中众所周知的数据稀疏问题的手段。我们的多样性分数可依据语言特征进行解释，并能够识别数据集中未被代表的语言类型。使用我们的方法，我们分析了一系列流行的多语言数据集（UD、Bible100、mBERT、XTREME、XGLUE、XNLI、XCOPA、TyDiQA、XQuAD）。除了对这些数据集进行排名之外，我们还发现，例如，（多式）综合语在几乎所有数据集中都缺失。
英文摘要：
Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. Linguistic diversity of these data sets is typically measured as the number of languages or language families included in the sample, but such measures do not consider structural properties of the included languages. In this paper, we propose assessing linguistic diversity of a data set against a reference language sample as a means of maximising linguistic diversity in the long run. We represent languages as sets of features and apply a version of the Jaccard index suitable for comparing sets of measures. In addition to the features extracted from typological data bases, we propose an automatic text-based measure, which can be used as a means of overcoming the well-known problem of data sparsity in manually collected features. Our diversity score is interpretable in terms of linguistic features and can identify the types of languages that are not represented in a data set. Using our method, we analyse a range of popular multilingual data sets (UD, Bible100, mBERT, XTREME, XGLUE, XNLI, XCOPA, TyDiQA, XQuAD). In addition to ranking these data sets, we find, for example, that (poly)synthetic languages are missing in almost all of them.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
