聘请你的人类学家!从人类学视角重新思考文化基准
Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens
AI总结:
针对当前大语言模型文化基准将文化静态化、同质化的问题,提出四部分分类框架,分析20个基准的6类方法缺陷,并借鉴人类学方法提出改进方向,以更准确评估模型的复杂文化响应能力。
摘要:
大语言模型的文化评估愈发重要,但当前基准常将文化简化为静态事实或同质化价值观。这一观点与强调文化具有动态性、历史情境性且在实践中生成的人类学论述相悖。为分析这一差距,我们提出一个四部分框架,对基准构建文化的方式进行分类,即知识、偏好、表现或偏见。借助这一视角,我们定性研究了20个文化基准,识别出6个反复出现的方法学问题,包括将国家等同于文化、忽视文化内部多样性,以及依赖过度简化的调查形式。借鉴成熟的人类学方法,我们提出了具体改进方案:纳入真实世界叙事与场景、让文化社群参与设计与验证,以及在情境中而非孤立地评估模型。我们旨在指导文化基准的开发,使其超越静态回忆任务,更准确地捕捉模型对复杂文化情境的响应。
英文摘要:
Cultural evaluation of large language models has become increasingly important, yet current benchmarks often reduce culture to static facts or homogeneous values. This view conflicts with anthropological accounts that emphasize culture as dynamic, historically situated, and enacted in practice. To analyze this gap, we introduce a four-part framework that categorizes how benchmarks frame culture, such as knowledge, preference, performance, or bias. Using this lens, we qualitatively examine 20 cultural benchmarks and identify six recurring methodological issues, including treating countries as cultures, overlooking within-culture diversity, and relying on oversimplified survey formats. Drawing on established anthropological methods, we propose concrete improvements: incorporating real-world narratives and scenarios, involving cultural communities in design and validation, and evaluating models in context rather than isolation. Our aim is to guide the development of cultural benchmarks that go beyond static recall tasks and more accurately capture the responses of the models to complex cultural situations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。