The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India's unique socio-cultural nuances. To bridge this gap, we introduce IndiBias, a comprehensive benchmarking dataset designed specifically for evaluating social biases in the Indian context. We filter and translate the existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian context in Hindi language. Additionally, we leverage LLMs including ChatGPT and InstructGPT to augment our dataset with diverse societal biases and stereotypes prevalent in India. The included bias dimensions encompass gender, religion, caste, age, region, physical appearance, and occupation. We also build a resource to address intersectional biases along three intersectional dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias measurement across different demographics. The dataset is available in English and Hindi, providing a size comparable to existing benchmark datasets. Furthermore, using IndiBias we compare ten different language models on multiple bias measurement metrics. We observed that the language models exhibit more bias across a majority of the intersectional groups.
IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments
Department of Data Science and Engineering, Indian Institute of Science Education and Research（数据科学与工程系，印度科学教育与研究学院）；Oracle Industries AI, Oracle Corporation（Oracle人工智能部，Oracle公司）
DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
Natural Language and Information Processing Lab (NLIP) Indian Institute of Technology Hyderabad（自然语言与信息处理实验室（NLIP）印度理工学院海得拉巴）
InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
Centre for Responsible AI, Indian Institute of Technology Madras（印度马德拉斯理工学院负责任人工智能中心）；International Institute of Information Technology, Hyderabad（海德拉巴国际信息技术学院）；AmexAI Labs, American Express（美国运通AmexAI实验室）；Wadhwani School of Data Science and AI, Indian Institute of Technology Madras（印度马德拉斯理工学院瓦德瓦尼数据科学与人工智能学院）
AMBEDKAR-A Multi-level Bias Elimination through a Decoding Approach with Knowledge Augmentation for Robust Constitutional Alignment of Language Models
Indian Institute of Information Technology, Kalyani（印度信息技术学院，卡里尼）；BITS Pilani Goa（比尔·斯图尔特学院，果阿）；IIT Madras（马德拉斯理工学院）；DTU（达丁理工大学）；Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；Meta AI；Amazon GenAI（亚马逊生成人工智能）
