DIWALI：多样性与包容性-aware的文化特定项目用于印度：数据集和对LLM进行文化文本适应的评估
DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
AI总结：
DIWALI数据集通过17个文化维度的8,000个文化概念，评估LLM在印度文化文本适应中的文化意识与一致性。
摘要：
大型语言模型（LLMs）被广泛用于各种任务和应用。然而，尽管它们具有广泛的能力，但它们被证明缺乏文化一致性[1,2]，并且由于缺乏文化知识和能力而产生有偏见的生成[3]。评估LLMs的文化意识和一致性尤其具有挑战性，因为缺乏适当的评估指标和无法获得代表区域和次区域复杂文化的文化扎根数据集。现有的文化特定项目（CSIs）数据集主要关注区域层面的概念，可能包含假阳性。为了解决这个问题，我们介绍了一个新的印度文化CSI数据集，属于17个文化方面。该数据集包含来自36个次区域的约8,000个文化概念。为了衡量LLMs在文化文本适应任务上的文化能力，我们使用创建的CSIs、LLM作为评判者以及来自不同社会人口区域的人类评估进行评估。此外，我们进行了定量分析，展示了所有考虑的LLMs在选择性次区域覆盖和表面级适应方面的表现。我们的数据集可在此处获取：https://huggingface.co/datasets/nlip/DIWALI，项目网页 https://nlip-lab.github.io/nlip/publications/diwali/，以及我们的代码库和模型输出可在此处找到：https://github.com/pramitsahoo/culture-evaluation
英文摘要：
Large language models (LLMs) are widely used in various tasks and applications. However, despite their wide capabilities, they are shown to lack cultural alignment \citep{ryan-etal-2024-unintended, alkhamissi-etal-2024-investigating} and produce biased generations \cite{naous-etal-2024-beer} due to a lack of cultural knowledge and competence. Evaluation of LLMs for cultural awareness and alignment is particularly challenging due to the lack of proper evaluation metrics and unavailability of culturally grounded datasets representing the vast complexity of cultures at the regional and sub-regional levels. Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. To address this issue, we introduce a novel CSI dataset for Indian culture, belonging to 17 cultural facets. The dataset comprises ~8k cultural concepts from 36 sub-regions. To measure the cultural competence of LLMs on a cultural text adaptation task, we evaluate the adaptations using the CSIs created, LLM as Judge, and human evaluations from diverse socio-demographic region. Furthermore, we perform quantitative analysis demonstrating selective sub-regional coverage and surface-level adaptations across all considered LLMs. Our dataset is available here: https://huggingface.co/datasets/nlip/DIWALI, project webpage https://nlip-lab.github.io/nlip/publications/diwali/, and our codebase with model outputs can be found here: https://github.com/pramitsahoo/culture-evaluation
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
