COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models
International Institute of Information Technology, Hyderabad(国际信息技术学院海得拉巴分校);Amazon GenAI(亚马逊生成式人工智能);University of Maryland Baltimore County(马里兰大学巴尔的摩县分校)
Large Language Models (LLMs) often inherit biases from the web data they are trained on, which contains stereotypes and prejudices. Current methods for evaluating and mitigating these biases rely on bias-benchmark datasets. These benchmarks measure bias by observing an LLM's behavior on biased statements. However, these statements lack contextual considerations of the situations they try to present. To address this, we introduce a contextual reliability framework, which evaluates model robustness to biased statements by considering the various contexts in which they may appear. We develop the Context-Oriented Bias Indicator and Assessment Score (COBIAS) to measure a biased statement's reliability in detecting bias, based on the variance in model behavior across different contexts. To evaluate the metric, we augmented 2,291 stereotyped statements from two existing benchmark datasets by adding contextual information. We show that COBIAS aligns with human judgment on the contextual reliability of biased statements (Spearman's $ρ= 0.65, p = 3.4 * 10^{-60}$) and can be used to create reliable benchmarks, which would assist bias mitigation works.
XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization
School of Computer Science and Informatics, Cardiff University, United Kingdom(英国卡迪夫大学计算机科学与信息学院);Tehran Institute for Advanced Studies, Iran(伊朗德黑兰高级研究所);Department of Digital Humanities, University of Helsinki, Finland(芬兰赫尔辛基大学数字人文系);SapienzaNLP Group, Computer Science Department, Sapienza University of Rome, Italy(意大利罗马大学计算机科学系萨皮恩扎NLP小组)
JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge
Institute of Science Tokyo, Japan(东京科学研究所,日本);ROIS-DS, Center for Juris-informatics, Japan(司法信息中心,日本);National Institute of Informatics, Japan(日本信息机构);University of Tokyo, Japan(东京大学,日本)
ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Microsoft(微软);ELLIS Institute Tübingen and MPI for Intelligent Systems(图宾根ELLIS研究所和智能系统研究所);Tübingen AI Center(图宾根人工智能中心)
Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes
MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
School of Electrical and Computer Engineering, College of Engineering, University of Tehran(德黑兰大学电气与计算机工程学院,工程学院);Tehran Institute for Advanced Studies, Khatam University(德黑兰高级研究 institute,Khatam 大学)
Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
Information and Processing Telecommunications Center(信息与处理电信中心);Universidad Politécnica de Madrid(马德里理工大学);Politecnico di Milano(米兰理工大学);Banco de España(西班牙银行)
Is Context Helpful for Chat Translation Evaluation?
Instituto de Telecomunicações(电信研究所);Unbabel(Unbabel公司);Instituto Superior Técnico(高等技术学院);Universidade de Lisboa(里斯本大学);Carnegie Mellon University(卡内基梅隆大学);INESC-ID(INESC-ID研究所)
Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim(曼海姆大学自然语言学习组(NLLG)商学院信息与数学学院);University of Technology Nuremberg (UTN), Department Engineering(纽伦堡技术大学(UTN)工程系);University of Gent, Department of Translation, Interpreting and Communication(根特大学翻译、口译与传播系);University of Aberdeen, Department of Computing Science(阿伯丁大学计算科学系);Natural Language Learning and Generation (NLLG) Lab(自然语言学习与生成(NLLG)实验室)
CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models
Zhejiang University(浙江大学);Westlake University(西湖大学);Department of Psychology, Jing Hengyi School of Education, Hangzhou Normal University(心理系,金恒毅教育学院,杭州师范大学)
AAAR-1.0: Assessing AI's Potential to Assist Research
Netflix(奈飞公司);University of California, Davis(加州大学戴维斯分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Ohio State University(俄亥俄州立大学)