arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13016cs.CL
收藏

理解多语言微调中语言特定类别不平衡的影响

Understanding the effects of language-specific class imbalance in multilingual fine-tuning

Idiap Research Institute(Idiap研究所)

Vincent Jung, Lonneke van der Plas

AI总结:

本文研究多语言分类中标签跨语言分布不均对LLM微调的影响,提出按语言分别计算类别权重的方法,有效缓解了性能下降和语言分离加剧等问题。

摘要:

我们研究了现实多语言分类数据集中常见的一类不平衡现象:标签在不同语言间分布不均。我们证明,在此类不平衡的数据集上微调基于Transformer的大型语言模型(LLM)会导致性能下降、潜在空间中语言分离更加明显,并促进无信息特征。我们改进了传统针对不平衡的类别加权方法,按每种语言分别计算类别权重,并表明这有助于缓解这些不利影响。这些结果使人们认识到多语言微调中语言特定类别不平衡的负面效应,以及模型学会依赖语言分离来完成任务的方式。

英文摘要:

We study the effect of one type of imbalance often present in real-life multilingual classification datasets: an uneven distribution of labels across languages. We show evidence that fine-tuning a transformer-based Large Language Model (LLM) on a dataset with this imbalance leads to worse performance, a more pronounced separation of languages in the latent space, and the promotion of uninformative features. We modify the traditional class weighing approach to imbalance by calculating class weights separately for each language and show that this helps mitigate those detrimental effects. These results create awareness of the negative effects of language-specific class imbalance in multilingual fine-tuning and the way in which the model learns to rely on the separation of languages to perform the task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑