arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.10946cs.CLcs.AIcs.LG
收藏

CultureLLM:将文化差异融入大型语言模型

CultureLLM: Incorporating Cultural Differences into Large Language Models

Institute of Software, CAS(中国科学院软件研究所);Microsoft Research(微软研究院)

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

AI总结:

本文提出 CultureLLM,利用 WVS 的 50 个种子样本进行语义数据增强并微调,以低成本让 LLM 建模 9 种文化差异,在 60 个数据集上显著超过 GPT-3.5 和 Gemini Pro。

摘要:

据报告,由于英语语料在训练数据中占据主导地位,大型语言模型(LLM)会偏向某些文化。由于多语言文化数据的收集成本通常很高,现有工作通过提示工程或特定文化预训练来处理这一问题。然而,这些方法可能忽视低资源文化的知识缺失,并且需要大量计算资源。本文提出 CultureLLM,这是一种将文化差异融入 LLM 的高性价比解决方案。CultureLLM 采用 World Value Survey(WVS,世界价值观调查)作为种子数据,并通过所提出的语义数据增强生成语义等价的训练数据。仅使用来自 WVS 的 50 个种子样本及增强数据,我们针对覆盖富资源语言和低资源语言的 9 种文化,微调了特定文化的 LLM 以及一个统一模型(CultureLLM-One)。在 60 个文化相关数据集上的大量实验表明,CultureLLM 显著优于 GPT-3.5(提升 8.1%)和 Gemini Pro(提升 9.5%)等多种同类模型,并取得与 GPT-4 相当甚至更好的性能。我们的人工研究表明,生成样本与原始样本在语义上等价,为 LLM 增强提供了有效解决方案。代码已发布于 https://github.com/Scarelette/CultureLLM。

英文摘要:

Large language models (LLMs) are reported to be partial to certain cultures owing to the training data dominance from the English corpora. Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training. However, they might overlook the knowledge deficiency of low-resource culture and require extensive computing resources. In this paper, we propose CultureLLM, a cost-effective solution to incorporate cultural differences into LLMs. CultureLLM adopts World Value Survey (WVS) as seed data and generates semantically equivalent training data via the proposed semantic data augmentation. Using only 50 seed samples from WVS with augmented data, we fine-tune culture-specific LLMs and one unified model (CultureLLM-One) for 9 cultures covering rich and low-resource languages. Extensive experiments on 60 culture-related datasets demonstrate that CultureLLM significantly outperforms various counterparts such as GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable performance to GPT-4 or even better. Our human study shows that the generated samples are semantically equivalent to the original samples, providing an effective solution for LLMs augmentation. Code is released at https://github.com/Scarelette/CultureLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑