CIVICS：构建用于审视大语言模型中文化知情价值观的数据集
CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models
AI总结：
该论文提出CIVICS数据集，用于评估大语言模型在多种语言和价值观敏感话题上的社会文化差异，通过动态标注和两种实验设置揭示模型响应中的拒绝模式与话题差异，促进AI的全球文化多样性和价值观多元性研究。
摘要：
本文介绍了“CIVICS：文化知情与价值观包容性社会影响语料库”数据集，旨在评估大语言模型（LLMs）在多种语言和价值观敏感话题上的社会与文化差异。我们创建了一个手工制作的多语言价值观负载提示数据集，涉及特定的社会敏感话题，包括LGBTQI权利、社会福利、移民、残障人士权利和代孕。CIVICS旨在生成能够展示LLMs编码和隐含价值观的响应。通过我们的动态标注流程、定制提示设计和实验，我们研究了开放权重LLMs如何回应价值观敏感问题，探索它们在多样语言和文化背景下的行为。利用基于对数概率和长形式响应的两种实验设置，我们展示了不同LLMs之间的社会和文化变异性。具体而言，涉及长形式响应的实验表明，拒绝在不同模型中被触发的情况各异，但在英语或翻译陈述中更一致且更频繁地出现。此外，特定话题和来源导致模型答案之间的差异更为显著，尤其是在移民、LGBTQI权利和社会福利方面。正如我们的实验所示，CIVICS数据集旨在作为未来研究的工具，促进更广泛语言环境下的可重复性和透明度，并进一步推动尊重和反映全球文化多样性和价值观多元性的人工智能技术的发展。CIVICS数据集和工具将在发表后以开放许可提供；匿名版本目前可在https://huggingface.co/CIVICS-dataset获取。
英文摘要：
This paper introduces the "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset, designed to evaluate the social and cultural variation of Large Language Models (LLMs) across multiple languages and value-sensitive topics. We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy. CIVICS is designed to generate responses showing LLMs' encoded and implicit values. Through our dynamic annotation processes, tailored prompt design, and experiments, we investigate how open-weight LLMs respond to value-sensitive issues, exploring their behavior across diverse linguistic and cultural contexts. Using two experimental set-ups based on log-probabilities and long-form responses, we show social and cultural variability across different LLMs. Specifically, experiments involving long-form responses demonstrate that refusals are triggered disparately across models, but consistently and more frequently in English or translated statements. Moreover, specific topics and sources lead to more pronounced differences across model answers, particularly on immigration, LGBTQI rights, and social welfare. As shown by our experiments, the CIVICS dataset aims to serve as a tool for future research, promoting reproducibility and transparency across broader linguistic settings, and furthering the development of AI technologies that respect and reflect global cultural diversities and value pluralism. The CIVICS dataset and tools will be made available upon publication under open licenses; an anonymized version is currently available at https://huggingface.co/CIVICS-dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
