生成式大型语言模型的丹麦语素养
Danoliteracy of Generative Large Language Models
AI总结：
针对丹麦语等低资源语言缺乏评估语料库的问题，提出评估丹麦语语言与文化能力的GLLM基准，发现模型排名与人类反馈高度相关且性能受单一潜在因子主导。
摘要：
生成式大型语言模型（GLLMs）的语言技术飞跃时刻不仅限于英语：这些模型也为低资源语言带来了技术应用的激增、投资和炒作。然而，由于缺乏适用的评估语料库，直到最近，这些模型在丹麦语等语言中的能力仍难以在定性演示之外进行验证。我们提出了一个 GLLM 基准来评估“Danoliteracy”，这是一种衡量丹麦语语言和文化能力的指标，涵盖八个不同场景，如丹麦公民身份测试和抽象社交媒体问答。我们发现这个有限规模的基准能够产生稳健的排名，与人类反馈的相关性为 $ρ\sim 0.8$，其中 GPT-4 和 Claude Opus 模型获得了最高排名。通过分析这些模型在各场景中的结果，我们发现一个强有力的潜在因素解释了 GLLMs 在丹麦语中 $95\\%$ 的场景性能方差，表明模型在语言适应方面存在一致性 $g$ 因子。
英文摘要：
The language technology moonshot moment of Generative Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments, and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were, until recently, difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate \emph{Danoliteracy}, a measure of Danish language and cultural competency across eight diverse scenarios such as Danish citizenship tests and abstractive social media question answering. This limited-size benchmark was found to produce a robust ranking that correlates to human feedback at $ρ\sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining $95\%$ of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
