As large language models (LLMs) have been used in many downstream tasks, the internal stereotypical representation may affect the fairness of the outputs. In this work, we introduce human knowledge into natural language interventions and study pre-trained language models' (LMs) behaviors within the context of gender bias. Inspired by CheckList behavioral testing, we present a checklist-style task that aims to probe and quantify LMs' unethical behaviors through question-answering (QA). We design three comparison studies to evaluate LMs from four aspects: consistency, biased tendency, model preference, and gender preference switch. We probe one transformer-based QA model trained on SQuAD-v2 dataset and one autoregressive large language model. Our results indicate that transformer-based QA model's biased tendency positively correlates with its consistency, whereas LLM shows the opposite relation. Our proposed task provides the first dataset that involves human knowledge for LLM bias evaluation.
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Tohoku University(东洋大学);Research and Development Center for Large Language Models, National Institute of Informatics(大型语言模型研究与开发中心,日本信息处理学会);National Institute of Informatics(日本信息处理学会)
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
Computer Science and Engineering, University of Michigan(计算机科学与工程系,密歇根大学);University of Michigan Law School(密歇根大学法学院);School of Information, University of Michigan(信息学院,密歇根大学);Materials Science & Engineering, University of Michigan(材料科学与工程系,密歇根大学);Department of Chemistry, Carnegie Mellon University(化学系,卡内基梅隆大学);Biomedical Engineering, University of Michigan(生物医学工程系,密歇根大学)
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Xi’an Jiaotong-Liverpool University(西交利物浦大学);University of Liverpool(利物浦大学);University of Macau(澳门大学);HKUST(香港科技大学);Microsoft Research Asia(微软亚洲研究院);Duke Kunshan University(昆山杜克大学)
Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models
Institute for Artificial Intelligence, Tsinghua University (THUAI)(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系);TMLR Group, Hong Kong Baptist University(香港浸会大学TMLR组)
SC-Taxo: Hierarchical Taxonomy Generation under Semantic Consistency Constraints using Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology Beijing(北京科技大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
Central South University(中南大学);Harbin Institute of Technology(哈尔滨工业大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Singapore Management University(新加坡管理大学)
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan(日本国家信息技术研究所);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国)
Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
Jožef Stefan Institute(乔塞夫·斯塔芬研究所);University of Ljubljana(卢布尔雅那大学);Faculty of Arts, University of Ljubljana(艺术学院,卢布尔雅那大学);Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Institute of Contemporary History(现代史研究所)
A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models
ELLIS Alicante(欧洲学习与智能系统实验室阿利坎特分部);Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague(捷克布拉格技术大学捷克信息学、机器人学与控制论研究所);Delft University of Technology(代尔夫特理工大学)