arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07302cs.CLcs.AIcs.LG
收藏

BertaQA:语言模型对本土文化了解多少?

BertaQA: How Much Do Language Models Know About Local Culture?

HiTZ Center – Ixa, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心 – Ixa)

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, Mikel Artetxe

AI总结:

研究引入英巴斯克双语问答数据集BertaQA,发现LLMs在本土文化知识上表现不佳,但巴斯克语持续预训练能实现知识向英语查询的转移。

摘要:

大型语言模型(LLMs)展现出关于世界的广泛知识,但大多数评估局限于全球或以英语为中心的主题。这引发了一个问题:这些模型在与网络存在感不那么显著的其他文化相关主题上表现如何。为了填补这一空白,我们引入了BertaQA,一个英语和巴斯克语平行的多项选择问答数据集。该数据集包含与巴斯克文化相关的本地子集,以及具有更广泛兴趣的全球子集。我们发现,最先进的LLMs在本土文化知识上表现挣扎,即使它们在全球主题上表现优异。然而,我们展示了在巴斯克语中进行持续预训练显著提升了模型在巴斯克文化上的表现,即使是在英语查询时也是如此。据我们所知,这是从低资源语言到高资源语言知识转移的第一个确凿证据。我们的分析揭示了语言与知识之间复杂的相互作用,并表明一些先前的发现在本土主题上重新评估时并不完全成立。我们的数据集和评估代码在开放许可下可在 https://github.com/juletx/BertaQA 获取。

英文摘要:

Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at https://github.com/juletx/BertaQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑