跨语言低资源形态分割中数据驱动的模型泛化
Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation
摘要:
模型评估的常见设计通常集中在单语设置,其中不同的模型根据它们在单个数据集上的性能进行比较,该数据集被假设为代表手头任务的所有可能数据。虽然这对于大型数据集可能是合理的,但在低资源情况下很难维持这种假设,其中数据汇集的工件可能产生离群值的数据集,从而可能使关于模型性能的结论一致。为了解决这些问题,我们研究模型的泛化能力在跨语言的低资源的情况下。使用形态分割作为测试用例,我们比较了三大类具有不同参数化的模型,从6个语言家族的11种语言中获取数据。在每个实验设置中,我们在第一个数据集上评估所有模型,然后在引入相同大小的新随机采样数据集以及将训练模型应用于不同大小的未知测试集时检查它们的性能一致性。结果表明,模型泛化的程度取决于数据集的特征,而不一定严重依赖于数据集的大小。在我们研究的特征中,训练集和测试集之间的词素重叠比率和每个词的平均词素数是两个最显著的因素。我们的研究结果表明,未来的工作应该采用随机抽样,以构建不同大小的数据集,以作出更负责任的索赔模型评估。
英文摘要:
Common designs of model evaluation typically focus on monolingual settings, where different models are compared according to their performance on a single data set that is assumed to be representative of all possible data for the task at hand. While this may be reasonable for a large data set, this assumption is difficult to maintain in low-resource scenarios, where artifacts of the data collection can yield data sets that are outliers, potentially making conclusions about model performance coincidental. To address these concerns, we investigate model generalizability in crosslinguistic low-resource scenarios. Using morphological segmentation as the test case, we compare three broad classes of models with different parameterizations, taking data from 11 languages across 6 language families. In each experimental setting, we evaluate all models on a first data set, then examine their performance consistency when introducing new randomly sampled data sets with the same size and when applying the trained models to unseen test sets of varying sizes. The results demonstrate that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data set size. Among the characteristics that we studied, the ratio of morpheme overlap and that of the average number of morphemes per word between the training and test sets are the two most prominent factors. Our findings suggest that future work should adopt random sampling to construct data sets with different sizes in order to make more responsible claims about model evaluation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。