评估大型语言模型中的波兰语言与文化能力
Evaluating Polish linguistic and cultural competency in large language models
AI总结：
针对大型语言模型缺乏文化背景知识易致细微错误的问题，提出包含600个手工制作问题的波兰语言与文化能力基准，涵盖六个类别，并评估了30多个模型，为模型波兰能力提供新视角。
摘要：
大型语言模型（LLMs）在处理和生成多语言文本方面日益熟练，这使其能够更有效地解决现实世界的问题。然而，语言理解是一个远比简单文本分析更复杂的问题。它需要熟悉文化背景，包括对日常生活、历史事件、传统、民俗、文学和流行文化的引用。缺乏此类知识可能导致误解以及难以察觉的细微错误。为了检验语言模型对波兰文化背景的知识，我们引入了波兰语言与文化能力基准，包含600个手工制作的问题。该基准分为六个类别：历史、地理、文化与传统、艺术与娱乐、语法和词汇。作为研究的一部分，我们进行了广泛的评估，涉及30多个开放权重和商业LLMs。我们的实验为语言模型中的波兰能力提供了新视角，超越了传统的自然语言处理任务和通用知识评估。
英文摘要：
Large language models (LLMs) are becoming increasingly proficient in processing and generating multilingual texts, which allows them to address real-world problems more effectively. However, language understanding is a far more complex issue that goes beyond simple text analysis. It requires familiarity with cultural context, including references to everyday life, historical events, traditions, folklore, literature, and pop culture. A lack of such knowledge can lead to misinterpretations and subtle, hard-to-detect errors. To examine language models' knowledge of the Polish cultural context, we introduce the Polish linguistic and cultural competency benchmark, consisting of 600 manually crafted questions. The benchmark is divided into six categories: history, geography, culture & tradition, art & entertainment, grammar, and vocabulary. As part of our study, we conduct an extensive evaluation involving over 30 open-weight and commercial LLMs. Our experiments provide a new perspective on Polish competencies in language models, moving past traditional natural language processing tasks and general knowledge assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
