Multiple-choice questions (MCQ) are frequently used to assess large language models (LLMs). Typically, an LLM is given a question and selects the answer deemed most probable after adjustments for factors like length. Unfortunately, LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to inherent biases of priori unbalanced probabilities, influencing the prediction of answers based on these IDs. Previous research has introduced methods to reduce this ''selection bias'' by simply permutating options on a few test samples and applying to new ones. Another problem of MCQ is the lottery ticket choice by ''random guessing''. The LLM does not learn particular knowledge, but the option is guessed correctly. This situation is especially serious for those small-scale LLMs. To address them, a more thorough approach involves shifting from MCQ to open-style questions, which can fundamentally eliminate selection bias and random guessing issues. However, transitioning causes its own set of challenges in (1) identifying suitable open-style questions and (2) validating the correctness of LLM open-style responses against human-annotated ground-truths. This work aims to tackle these significant difficulties, and establish a new LLM evaluation benchmark through entirely open-style questions. Consequently, we introduce the Open-LLM-Leaderboard to track various LLMs' performance and reflect true capability of them, such as GPT-4o/4/3.5, Claude 3, Gemini, etc. Our code and dataset are available at https://github.com/VILA-Lab/Open-LLM-Leaderboard.
Cohere Labs（Cohere 实验室）；Cohere（Cohere 公司）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；University of Waterloo（滑铁卢大学）；Massachusetts Institute of Technology（麻省理工学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
Drawing Conclusions from Draws: Rethinking Preference Semantics in Arena-Style LLM Evaluation
Centre for Artificial Intelligence, University College London（人工智能中心，伦敦大学学院）；University of Waterloo（滑铁卢大学）；University of Copenhagen（哥本哈根大学）；Research and Development Center for Large Language Models, National Institute of Informatics（大语言模型研究与发展中心，日本信息处理学会）
Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability
LibrAI；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Oracle（甲骨文公司）；The University of Melbourne（墨尔本大学）；Tsinghua University（清华大学）；Princeton University（普林斯顿大学）；Peking University（北京大学）；CUHK（香港中文大学）；BUPT（北京邮电大学）；UCSC（加利福尼亚大学圣克鲁兹分校）；SUTD（新加坡科技设计大学）；University of Edinburgh（爱丁堡大学）；Concordia AI；NTU（南洋理工大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；Shanghai Jiao Tong University（上海交通大学）
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
National University of Singapore（国立新加坡大学）；Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理学院）；Singapore University of Technology and Design（新加坡科技设计大学）
Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models
Tallinn University of Technology（塔林技术大学）；MBZUAI（穆罕默德·本·拉希德·阿勒马克图姆智能研究院）；Idiap Research Institute（伊迪普研究 institute）；Univ. Rennes, CNRS, Irisa（里昂大学，CNRS，Irisa）；Univ. Bretagne Sud, CNRS, Irisa（布列塔尼南部大学，CNRS，Irisa）；Validsoft Ltd.（Validsoft有限公司）
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Shenzhen Loop Area Institute（深圳河套学院）；CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学CentraleSupélec分校）；National University of Singapore（新加坡国立大学）
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
Salesforce AI Research（Salesforce AI研究院）；Hong Kong Baptist University（香港浸会大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；The Australian National University（澳大利亚国立大学）
Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系）；Salesforce AI Research（Salesforce AI 研究院）；The Hong Kong University of Science and Technology（香港科学与技术大学）；National University of Singapore（新加坡国立大学）；The Education University of Hong Kong（香港教育大学）
BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）；Mayo Clinic（梅奥诊所）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Harvard University（哈佛大学）；Stanford University（斯坦福大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Kempner Institute for the Study of Natural and Artificial Intelligence（肯普纳自然与人工智能研究所）；Broad Institute of MIT and Harvard（博德研究所）；Harvard Data Science Initiative（哈佛数据科学计划）
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院）；Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系）；Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院）；School of Humanities, Tallinn University（塔林大学人文学院）
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Technical University of Munich (TUM)（慕尼黑技术大学）；Ludwig Maximilian University of Munich (LMU)（慕尼黑路德维希-马克西米利安大学）；University of Konstanz（康斯坦茨大学）；University of Saarbrücken（萨尔布吕肯大学）
WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Princess Margaret Cancer Centre, University Health Network（大学健康网络玛格丽特公主癌症中心）；Lunenfeld-Tanenbaum Research Institute, Sinai Health System（西奈健康系统勒嫩菲尔德-塔南鲍姆研究所）；Peter Munk Cardiac Centre, University Health Network（大学健康网络彼得·芒克心脏中心）；AI Hub, University Health Network（大学健康网络人工智能中心）
本文介绍了WangLab参加MEDIQA-M3G 2024共享任务的两个方案，分别基于Claude 3 Opus API调用和CLIP风格图像分类，在英语类别中取得第一和第二名，并指出这两种方法在医学视觉问答中具有进一步研究潜力。
