CulturalBench:通过人机文化协作构建的稳健、多样且具有挑战性的文化基准
CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming
AI总结:
本文提出 CulturalBench,一个包含 1,696 道人工编写并验证、覆盖 45 个地区与 17 个主题的文化知识基准,发现前沿模型准确率远低于人类,且在多正确答案问题上存在过拟合,其中 GPT-4o 跨文化表现最佳。
摘要:
稳健、多样且具有挑战性的文化知识基准对于衡量我们在构建能够跨多元文化提供帮助的语言模型方面的进展至关重要。我们提出了 CulturalBench:一组包含 1,696 道由人类编写并经人类验证的问题,用于评估语言模型的文化知识,覆盖 45 个全球地区,包括孟加拉国、津巴布韦和秘鲁等代表性不足的地区。每道问题均由五名独立标注者验证,并涵盖从饮食偏好到问候礼仪等 17 个多样化主题。我们采用受 Human-AI Red-Teaming 启发的方法构建 CulturalBench。与人类表现(92.4% 准确率)相比,CulturalBench 的困难版本即使对表现最好的前沿语言模型也具有挑战性,准确率从 28.7% 到 61.5% 不等。我们发现语言模型常常在具有多个正确答案的棘手问题上遇到困难(例如,中国人通常使用什么餐具?),暴露出过度拟合单一答案的倾向。我们的结果表明,GPT-4o 在跨文化表现上显著优于其他模型,超过了本地提供商(例如,Mistral 在欧洲文化上、DeepSeek 在中国文化上的表现)。总体而言,模型在与北非、南美和中东相关的问题上表现不佳。
英文摘要:
Robust, diverse, and challenging cultural knowledge benchmarks are essential for measuring our progress towards making LMs that are helpful across diverse cultures. We introduce CulturalBench: a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. Questions are each verified by five independent annotators and span 17 diverse topics ranging from food preferences to greeting etiquette. We construct CulturalBench using methods inspired by Human-AI Red-Teaming. Compared to human performance (92.4% accuracy), the hard version of CulturalBench is challenging even for the best-performing frontier LMs, ranging from 28.7% to 61.5% in accuracy. We find that LMs often struggle with tricky questions that have multiple correct answers (e.g., What utensils do the Chinese usually use?), revealing a tendency to overfit to a single answer. Our results indicate that GPT-4o substantially outperform other models across cultures, besting local providers (e.g., Mistral on European culture and DeepSeek on Chinese culture). Across the board, models under-perform on questions related to North Africa, South America and Middle East.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。