arXiv 2407.098232025-05-30cs.CLcs.AI

NativQA：面向LLM的多语言文化对齐自然查询

NativQA: Multilingual Culturally-Aligned Natural Query for LLMs

University of New Brunswick（新不伦瑞克大学）；Qatar Computing Research Institute（卡塔尔计算研究中心）；UPES（UPES大学）；Carnegie Mellon University in Qatar（卡塔尔卡内基梅隆大学）；Qatar University（卡塔尔大学）

Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, Firoj Alam

备注：: LLMs, Native, Multilingual, Language Diversity, Contextual Understanding, Minority Languages, Culturally Informed, Foundation Models, Large Language Models

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出可扩展且独立于语言的NativQA框架，构建文化与区域对齐的母语QA数据集，并基于9个地区7种语言约64k标注对构建MultiNativQA基准，用于评估和微调LLM。

摘要：

自然问答（QA）数据集在评估大型语言模型（LLM）的能力方面发挥着至关重要的作用，确保其在实际应用中的有效性。尽管已经开发了众多QA数据集并开展了一些并行工作，但仍然明显缺乏一个框架以及由母语用户用自己的语言查询的大规模特定区域数据集。这一空白阻碍了针对区域和文化特性的有效基准测试和微调模型的开发。在本研究中，我们提出了一个可扩展的、独立于语言的框架NativQA，以无缝构建母语的文化和区域对齐QA数据集，用于LLM评估和调整。我们通过设计一个多语言自然QA数据集MultiNativQA来证明所提框架的有效性，该数据集包含约64k个手动标注的QA对，涵盖七种语言（从高资源到极低资源），基于来自9个地区的母语者的查询，覆盖18个主题。我们使用MultiNativQA数据集对开源和闭源LLM进行了基准测试。我们已将MultiNativQA数据集(https://huggingface.co/datasets/QCRI/MultiNativQA)及其他实验脚本(https://gitlab.com/nativqa/multinativqa)向社区公开发布。

英文摘要：

Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。