Demographic classification is essential in fairness assessment in recommender systems or in measuring unintended bias in online networks and voting systems. Important fields like education and politics, which often lay a foundation for the future of equality in society, need scrutiny to design policies that can better foster equality in resource distribution constrained by the unbalanced demographic distribution of people in the country.
We collect three publicly available datasets to train state-of-the-art classifiers in the domain of gender and caste classification. We train the models in the Indian context, where the same name can have different styling conventions (Jolly Abraham/Kumar Abhishikta in one state may be written as Abraham Jolly/Abishikta Kumar in the other). Finally, we also perform cross-testing (training and testing on different datasets) to understand the efficacy of the above models.
We also perform an error analysis of the prediction models. Finally, we attempt to assess the bias in the existing Indian system as case studies and find some intriguing patterns manifesting in the complex demographic layout of the sub-continent across the dimensions of gender and caste.
Named Entity Recognition (NER) is often the first step towards automated Knowledge Base (KB) generation from raw text. In this work, we assess the bias in various Named Entity Recognition (NER) systems for English across different demographic groups with synthetically generated corpora. Our analysis reveals that models perform better at identifying names from specific demographic groups across two datasets. We also identify that debiased embeddings do not help in resolving this issue. Finally, we observe that character-based contextualized word representation models such as ELMo results in the…
Gender information is no longer a mandatory input when registering for an account at many leading Internet companies. However, prediction of demographic information such as gender and age remains an important task, especially in intervention of unintentional gender/age bias in recommender systems. Therefore it is necessary to infer the gender of those users who did not to provide this information during registration. We consider the problem of predicting the gender of registered users based on their declared name. By analyzing the first names of 100M+ users, we found that genders can be very e…
The increasing adoption of facial processing systems in India is fraught with concerns of privacy, transparency, accountability, and missing procedural safeguards. At the same time, we also know very little about how these technologies perform on the diverse features, characteristics, and skin tones of India's 1.34 billion-plus population. In this paper, we test the face detection and facial analysis functions of four commercial facial processing tools on a dataset of Indian faces. The tools display varying error rates in the face detection and gender and age classification functions. The gend…
We linked names and contact information to publicly available profiles in the Personal Genome Project. These profiles contain medical and genomic information, including details about medications, procedures and diseases, and demographic information, such as date of birth, gender, and postal code. By linking demographics to public records such as voter lists, and mining for names hidden in attached documents, we correctly identified 84 to 97 percent of the profiles for which we provided names. Our ability to learn their names is based on their demographics, not their DNA, thereby revisiting an …
IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments
Department of Data Science and Engineering, Indian Institute of Science Education and Research（数据科学与工程系，印度科学教育与研究学院）；Oracle Industries AI, Oracle Corporation（Oracle人工智能部，Oracle公司）
Towards Harmless Rawlsian Fairness Regardless of Demographic Prior
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research（新加坡科技研究局前沿人工智能研究中心）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Making
Indian Institute of Information Technology, Allahabad (IIITA)（印度阿勒颇理工学院，阿勒颇（IIITA））；National Institute of Electronics and Information Technology (NIELIT)（国家电子与信息技术研究所（NIELIT））
Social media data provides propitious opportunities for public health research. However, studies suggest that disparities may exist in the representation of certain populations (e.g., people of lower socioeconomic status). To quantify and address these disparities in population representation, we need demographic information, which is usually missing from most social media platforms. Here, we propose an ensemble approach for inferring demographics from social media data. Several methods have been proposed for inferring demographic attributes such as, age, gender and race/ethnicity. However, mo…
Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
World Bank（世界银行）；University of Oxford（牛津大学）；New York University（纽约大学）；University of Pennsylvania（宾夕法尼亚大学）；LMU Munich（慕尼黑大学）；Allen Institute for AI（人工智能研究院）
Most proposed algorithmic fairness techniques require access to data on a "sensitive attribute" or "protected category" (such as race, ethnicity, gender, or sexuality) in order to make performance comparisons and standardizations across groups, however this data is largely unavailable in practice, hindering the widespread adoption of algorithmic fairness. Through this paper, we consider calls to collect more data on demographics to enable algorithmic fairness and challenge the notion that discrimination can be overcome with smart enough technical methods and sufficient data alone. We show how …
Small Changes, Big Impact: Demographic Bias in LLM-Based Hiring Through Subtle Sociocultural Markers in Anonymised Resumes
Singapore University of Technology and Design (SUTD)（新加坡科技设计大学）；GovTech Singapore（新加坡政府科技局）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Agency for Science, Technology and Research (A*STAR)（科学技术研究局）；University of British Columbia (UBC)（不列颠哥伦比亚大学）
Computational social scientists often harness the Web as a "societal observatory" where data about human social behavior is collected. This data enables novel investigations of psychological, anthropological and sociological research questions. However, in the absence of demographic information, such as gender, many relevant research questions cannot be addressed. To tackle this problem, researchers often rely on automated methods to infer gender from name information provided on the web. However, little is known about the accuracy of existing gender-detection methods and how biased they are a…
