arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.04523cs.CL
收藏

当前 NLI 系统能够处理德语词序吗?考察语言模型在一个新的德语最小对立体挑战集上的表现

Can current NLI systems handle German word order? Investigating language model performance on a new German challenge set of minimal pairs

Mannheim University(曼海姆大学);Heidelberg University(海德堡大学)

Ines Reinig, Katja Markert

AI总结:

本文提出首个德语词序对抗 NLI 数据集 WOGLI,利用仅词序和形态变化不同的最小对立体,评估并揭示德语模型在翻译数据训练后处理词序推理的局限。

摘要:

与英语相比,德语词序更自由,因此给自然语言推理(NLI)带来了额外挑战。我们构建了 WOGLI(Word Order in German Language Inference,德语语言推理中的词序),这是首个针对德语词序的对抗性 NLI 数据集,具有以下特性:(i) 每个前提都有一个被蕴涵的假设和一个不被蕴涵的假设;(ii) 前提与假设之间仅在词序以及为标记格和数而必需的形态变化上有所不同。具体而言,每个前提及其两个假设包含完全相同的词元。我们的对抗样本要求模型利用形态标记来识别或拒绝蕴涵。我们表明,当前在翻译得到的 NLI 数据上微调的德语自编码模型在该挑战集上可能表现困难,这反映出翻译得到的 NLI 数据集不会再现目标语言中所有必要的语言现象。我们还考察了数据增强后的表现,以及在由 WOGLI 派生的相关词序现象上的表现。我们的数据集已公开发布于 https://github.com/ireinig/wogli。

英文摘要:

Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise andits two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at https://github.com/ireinig/wogli.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑