CaLMQA：探索横跨23种语言的文化特定长式问答
CaLMQA: Exploring culturally specific long-form question answering across 23 languages
AI总结：
本文构建涵盖23种语言、51.7K个文化特定问题的 CaLMQA 数据集，评估LLM长式答案，揭示低资源语言表层错误及文化问题事实错误更突出。
摘要：
尽管大语言模型（LLM）在全球的使用不断增长，但在许多语言中，它们针对文化特定问题生成长式答案的能力仍未得到探索。为填补这一空白，我们通过创建 CaLMQA 开展了首个文本多语言长式 QA 研究，该数据集包含横跨23种不同语言的51.7K个文化特定问题。我们将文化特定问题定义为涉及一种或少数几种文化所独有概念的问题，或者其答案会随文化或地区背景而不同的问题。我们通过从高资源语言的社区网络论坛中爬取自然出现的问题，并聘请母语使用者为斐济语和基隆迪语等资源不足、很少被研究的语言撰写问题，从而获得这些问题。我们的数据收集方法不依赖翻译，因此能够收集具有文化独特性的问题，例如“Kuber iki umwami wa mbere w'uburundi yitwa Ntare?”（基隆迪语；英文翻译为：“Why was the first king of Burundi called Ntare (Lion)?”，即“为什么布隆迪的第一位国王被称为 Ntare（狮子）？”）。我们评估了 LLM 生成长式答案的事实性、相关性和表层质量，发现：（1）对于许多语言，即使是最好的模型也会犯严重的表层错误（例如用错误语言回答、重复），尤其是在低资源语言上；（2）与文化无关问题（即在许多文化中具有一致含义和答案的问题）的答案相比，文化特定问题的答案包含更多事实错误。我们发布 CaLMQA，以促进未来关于文化和多语言长式 QA 的研究。
英文摘要：
Despite rising global usage of large language models (LLMs), their ability to generate long-form answers to culturally specific questions remains unexplored in many languages. To fill this gap, we perform the first study of textual multilingual long-form QA by creating CaLMQA, a dataset of 51.7K culturally specific questions across 23 different languages. We define culturally specific questions as those that refer to concepts unique to one or a few cultures, or have different answers depending on the cultural or regional context. We obtain these questions by crawling naturally-occurring questions from community web forums in high-resource languages, and by hiring native speakers to write questions in under-resourced, rarely-studied languages such as Fijian and Kirundi. Our data collection methodologies are translation-free, enabling the collection of culturally unique questions like "Kuber iki umwami wa mbere w'uburundi yitwa Ntare?" (Kirundi; English translation: "Why was the first king of Burundi called Ntare (Lion)?"). We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers, finding that (1) for many languages, even the best models make critical surface-level errors (e.g., answering in the wrong language, repetition), especially for low-resource languages; and (2) answers to culturally specific questions contain more factual errors than answers to culturally agnostic questions -- questions that have consistent meaning and answer across many cultures. We release CaLMQA to facilitate future research in cultural and multilingual long-form QA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
