arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17302cs.CL
收藏

大语言模型能否生成具有文化相关性的常识问答数据?以印度尼西亚语和巽他语为例的案例研究

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

KAIST(韩国科学技术院)

Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, Alice Oh

AI总结:

本研究针对低资源语言文化相关常识问答数据生成问题,用LLM结合人工构建了印度尼西亚语、巽他语各约4500条的最大同类数据集,发现直接生成法效果优于英文数据适配法,同时指出低资源语言存在流畅性错误多、文化深度不足的问题。

摘要:

大语言模型(LLM)正越来越多地被用于生成合成数据,以供模型训练和评估使用。然而,目前尚不清楚它们能否生成高质量的问答(QA)数据集,且该数据集需融入嵌入在语言中的知识与文化细微差别,针对低资源语言而言尤其如此。\n在本研究中,我们探究了利用LLM为印度尼西亚语和巽他语生成具有文化相关性的常识问答数据集的有效性。为此,我们采用涉及LLM和人类标注员的多种方法,为这两种语言创建了数据集,最终每种语言约有4500个问题(总计约9000个),使我们的数据集成为同类中规模最大的。\n我们的实验表明,对现有英文数据集进行自动数据适配的方法对巽他语效果较差。有意思的是,在目标语言上使用直接生成方法时,GPT-4 Turbo能够在两种语言中生成具备充足常识的问题,不过其文化“深度”不及人类生成的内容。我们还观察到,巽他语数据集中的流畅性错误出现率更高,凸显出中等资源语言与更低资源语言之间的差异。

英文摘要:

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ~4.5K questions per language (~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑