SANSKRITI：一个用于评估语言模型印度文化知识的综合基准
SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
AI总结：
本研究推出涵盖21853个问答对、覆盖印度28个邦与8个联邦属地16类文化维度的基准SANSKRITI，评测发现各类语言模型在印度文化细微查询尤其是区域特定语境上表现差异显著，为提升语言模型文化理解能力设立了新评测标准。
摘要：
语言模型（LM）是塑造现代工作流的不可或缺的工具，但其全球有效性取决于对当地社会文化语境的理解。为解决这一问题，我们推出SANSKRITI，这是一个专为评估语言模型对印度丰富文化多样性的理解能力而设计的基准。SANSKRITI包含21853对精心整理的问答对，覆盖28个邦和8个联邦属地，是目前用于测试印度文化知识的最大规模数据集。它涵盖印度文化的16个核心维度：仪式与典礼、历史、旅游、美食、舞蹈与音乐、服饰、语言、艺术、节日、宗教、医药、交通、体育、夜生活以及知名人物，全面呈现了印度的文化图景。我们在主流大语言模型（LLM）、印度语言模型（ILM）和小语言模型（SLM）上对SANSKRITI进行了评测，结果显示这些模型在处理文化细微差别的查询时能力存在显著差异，许多模型在特定区域语境的任务上表现不佳。通过提供一个规模庞大、文化内涵丰富且具备多样性的数据集，SANSKRITI为评估和提升语言模型的文化理解能力设立了新的标准。
英文摘要：
Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
