面向波罗的海与北欧语言的多语种大语言模型评估：立陶宛历史研究
Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History
AI总结：
本研究通过多语种立陶宛历史与通用历史选择题数据集，评估多款多语种及北欧微调大语言模型的表现，发现GPT-4o表现最优，仅靠文化历史关联无法提升模型性能。
摘要：
本研究在多项选择题问答任务中，评估了多语种大语言模型（LLM）的立陶宛历史知识与通用历史知识。研究使用的数据集包含立陶宛本国历史与通用历史问题，这些问题被译为波罗的海语言、北欧语言及其他语言（英语、乌克兰语、阿拉伯语），用于测试模型在文化与历史关联群体间的知识共享能力。本次评估的模型包括GPT-4o、LLaMa3.1 8b与70b、QWEN2.5 7b与72b、Mistral Nemo 12b、LLaMa3 8b、Mistral 7b、LLaMa3.2 3b，以及经过北欧语言微调的模型（GPT-SW3和LLaMa3 8b）。\n 结果显示，GPT-4o在所有语言组中表现始终优于其他所有模型，在波罗的海与北欧语言上的表现略优。QWEN2.5 72b、LLaMa3.1 70b等较大的开源模型表现良好，但在波罗的海语言上的对齐能力较弱。Mistral Nemo 12b、LLaMa3.2 3b、QWEN 7B、LLaMa3.1 8B、LLaMa3 8b等较小模型在波罗的海语言的立陶宛相关内容对齐上存在差距，但在北欧语言及其他语言上表现更好。经过北欧语言微调的模型并未超过多语种模型，这表明仅靠共享的文化或历史背景并不能保证更好的性能。
英文摘要：
In this work, we evaluated Lithuanian and general history knowledge of multilingual Large Language Models (LLMs) on a multiple-choice question-answering task. The models were tested on a dataset of Lithuanian national and general history questions translated into Baltic, Nordic, and other languages (English, Ukrainian, Arabic) to assess the knowledge sharing from culturally and historically connected groups. We evaluated GPT-4o, LLaMa3.1 8b and 70b, QWEN2.5 7b and 72b, Mistral Nemo 12b, LLaMa3 8b, Mistral 7b, LLaMa3.2 3b, and Nordic fine-tuned models (GPT-SW3 and LLaMa3 8b). Our results show that GPT-4o consistently outperformed all other models across language groups, with slightly better results for Baltic and Nordic languages. Larger open-source models like QWEN2.5 72b and LLaMa3.1 70b performed well but showed weaker alignment with Baltic languages. Smaller models (Mistral Nemo 12b, LLaMa3.2 3b, QWEN 7B, LLaMa3.1 8B, and LLaMa3 8b) demonstrated gaps with LT-related alignment with Baltic languages while performing better on Nordic and other languages. The Nordic fine-tuned models did not surpass multilingual models, indicating that shared cultural or historical context alone does not guarantee better performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
