评估大语言模型中的形态组合泛化能力
Evaluating Morphological Compositional Generalization in Large Language Models
AI总结：
本文通过组合性视角系统研究大语言模型在黏着语中的形态泛化能力，设计生成式与判别式任务评估多个先进多语言模型，发现其在面对新词根和复杂形态时性能显著下降且缺乏系统性。
摘要：
大语言模型（LLMs）在各种自然语言生成与理解任务中已展现出显著进展。然而，它们的语言泛化能力仍然存疑，令人怀疑这些模型是否以类似于人类的方式学习语言。人类在语言使用中表现出组合泛化和语言创造力，但LLMs在多大程度上复现这些能力——尤其是在形态学方面——仍未被充分探索。在本工作中，我们通过组合性的视角系统地研究LLMs的形态泛化能力。我们将语素定义为组合原语，并设计了一套新颖的生成式和判别式任务，以评估形态生产力和系统性。聚焦于土耳其语和芬兰语等黏着语，我们评估了多个最先进的指令微调多语言模型，包括GPT-4和Gemini。我们的分析表明，LLMs在形态组合泛化方面存在困难，尤其是在应用于新词根时，随着形态复杂度的增加，性能急剧下降。虽然模型识别单个形态组合的能力优于随机水平，但其表现缺乏系统性，导致与人类相比存在显著的准确率差距。
英文摘要：
Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
