A Novel Interpretability Metric for Explaining Bias in Language Models: Applications on Multilingual Models from Southeast Asia
School of Computer Science, University of Birmingham(英国伯明翰大学计算机科学学院);Department of Information Systems and Computer Science, Ateneo de Manila University(马尼拉大学信息系统与计算机科学系)
Lance Calvin Lim Gamboa, Mark Lee
备注:
Accepted for oral presentation at PACLIC 38 (38th Pacific Asia Conference on Language, Information, and Computation)
Work on bias in pretrained language models (PLMs) focuses on bias evaluation and mitigation and fails to tackle the question of bias attribution and explainability. We propose a novel metric, the $\textit{bias attribution score}$, which draws from information theory to measure token-level contributions to biased behavior in PLMs. We then demonstrate the utility of this metric by applying it on multilingual PLMs, including models from Southeast Asia which have not yet been thoroughly examined in bias evaluation literature. Our results confirm the presence of sexist and homophobic bias in Southeast Asian PLMs. Interpretability and semantic analyses also reveal that PLM bias is strongly induced by words relating to crime, intimate relationships, and helping among other discursive categories, suggesting that these are topics where PLMs strongly reproduce bias from pretraining data and where PLMs should be used with more caution.
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);Microsoft Research Asia (MSRA)(微软亚洲研究院);Sungkyunkwan University (SKKU)(成均馆大学);Universiti Brunei Darussalam (UBD)(文莱大学);China University of Petroleum (East China)(中国石油大学(华东));Nanyang Technological University (NTU)(南洋理工大学);Kyoto University(京都大学);Konkuk University(建国大学);Upstage AI
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Georgia Institute of Technology(佐治亚理工学院);Samsung R&D Institute Philippines(三星菲律宾研发院);Sungkyunkwan University(成均馆大学);Tohoku University(东北大学);National University of Singapore(新加坡国立大学);University of Copenhagen(哥本哈根大学);University of Michigan(密歇根大学);Indian Institute of Technology Delhi(印度理工学院德里);Institute of Science Tokyo(东京科学研究院)
SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages
SEACrowd;University of Indonesia(印度尼西亚大学);Cohere Labs Community(Cohere Labs 社区);Technical University of Denmark(丹麦技术大学);Chulalongkorn University(朱拉隆功大学);Vidyasirimedhi Institute of Science and Technology(维迪亚西里梅迪科学技术研究所);AI Singapore(新加坡人工智能)
MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
Singapore University of Technology and Design(新加坡科技设计大学);Agency for Science, Technology and Research, Singapore(新加坡科技研究局);Indian Institute of Technology Delhi(印度理工学院德里分校);Alibaba DAMO Academy(阿里巴巴达摩院);Microsoft Research Asia(微软亚洲研究院);Shanghai University of Finance and Economics(上海财经大学);Inner Mongolia University(内蒙古大学);Kyoto University(京都大学);Jiangxi Normal University(江西师范大学);Korea University(韩国大学);Nanyang Technological University(南洋理工大学)
Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages
Brown University(布朗大学);UC Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);HKUST(香港科技大学);Bloomberg(彭博);EleutherAI;Samsung R&D Institute Philippines(三星菲律宾研发中心);Stanford University(斯坦福大学);National University of Singapore(新加坡国立大学);VietAI Research(VietAI研究院);University of Potsdam(波茨坦大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
China-ASEAN Information Harbor Co., Ltd.(中国-东盟信息港有限公司);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Software & Microelectronics, Peking University(北京大学软件与微电子学院)
SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?
Vidyasirimedhi Institute of Science and Technology(威差亚西里梅迪科学技术研究所);AI Singapore(新加坡人工智能);King Mongkut’s Institute of Technology Ladkrabang(拉卡邦先皇理工学院);Nanyang Technological University(南洋理工大学);Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University(朱拉隆功大学工程学院计算机工程系)
Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia
School of Computer Science, University of Birmingham(英国伯明翰大学计算机科学学院);Department of Information Systems and Computer Science, Ateneo de Manila University(马尼拉亚托大学信息系统与计算机科学系)
SAGE: Sustainable Agent-Guided Expert-tuning for Culturally Attuned Translation in Low-Resource Southeast Asia
Xi'an Jiaotong-Liverpool University(西安交通大学利物浦大学);University of Liverpool(利物浦大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan
Center for Integrated Japanese Studies, Tohoku University(东京东京大学综合日语研究机构);University of California, Berkeley(加州大学伯克利分校);Berkeley Artificial Intelligence Research (BAIR)(伯克利人工智能研究(BAIR))
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment
Technical University of Munich(慕尼黑工业大学);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration
Department of EECS, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA(电子工程与科学系,埃姆布里-里德尔航空大学,日落海滩,佛罗里达州,美国);NLP Engineering Team, Hishab Singapore Pte. Ltd, Singapore(自然语言处理工程团队,Hishab新加坡私人有限公司,新加坡);Department of Computer Information Systems, Texas A&M University - Central Texas, Texas, USA(计算机信息系统系,德克萨斯A&M大学-中央德克萨斯分校,德克萨斯州,美国)
Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions
Computer Science and Engineering, BRAC University(布拉克大学计算机科学与工程系);Department of Computer Science and Engineering, BRAC University(布拉克大学计算机科学与工程系);Engineering, BRAC University(布拉克大学工程系)
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages
AI Singapore(新加坡人工智能中心);IndoNLP(印尼自然语言处理研究组);Universitas Indonesia(印度尼西亚大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);RevComm, Inc.(RevComm公司);Tohoku University(东北大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Bath(巴斯大学);National University Philippines(菲律宾国立大学);MOH Office for Healthcare Transformation (MOHT)(卫生部医疗转型办公室);NAIST(奈良科学技术研究所);Works Applications Lab(Works Applications实验室);Universitas Gadjah Mada(加查马达大学);Institut Teknologi Bandung(万隆理工学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);Institute for Infocomm Research, A*STAR(新加坡资讯通信研究院(新加坡科技研究局下属));Samsung Research Philippines(三星菲律宾研究院);University of Cambridge(剑桥大学);Queensland University of Technology(昆士兰科技大学)