对外评估大型语言模型中的文化素养
Extrinsic Evaluation of Cultural Competence in Large Language Models
AI总结：
本文通过两个文本生成任务评估模型在文化敏感性方面的表现，发现文化提示对输出有影响，但不同国家输出的相似性与文化价值观无显著关联。
摘要：
多样化的用户与语言技术之间的互动要求后者输出具有文化相关性和敏感性。先前的工作评估了模型对文化规范、价值观和物品的知识，但未考虑这种知识如何在下游应用中体现。在本工作中，我们专注于两个文本生成任务的外在评估：开放性问答和故事生成。我们定量和定性地评估当提示中明确提示文化，特别是国籍时，模型输出的变化。尽管我们发现当改变国籍和特征文化相关词汇时，模型输出确实有所变化，但我们还发现不同国家输出的相似性与这些国家的文化价值观之间存在弱相关性。最后，我们讨论了在面向用户的任务中设计全面评估文化能力的重要考虑因素。
英文摘要：
Productive interactions between diverse users and language technologies require outputs from the latter to be culturally relevant and sensitive. Prior works have evaluated models' knowledge of cultural norms, values, and artifacts, without considering how this knowledge manifests in downstream applications. In this work, we focus on extrinsic evaluation of cultural competence in two text generation tasks, open-ended question answering and story generation. We quantitatively and qualitatively evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. Although we find that model outputs do vary when varying nationalities and feature culturally relevant words, we also find weak correlations between text similarity of outputs for different countries and the cultural values of these countries. Finally, we discuss important considerations in designing comprehensive evaluation of cultural competence in user-facing tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
