Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh（计算机科学与工程系，伊斯兰技术大学，加兹ipur，孟加拉国）；Department of Computer Science（计算机科学系）；Engineering, Islamic University of Technology, Gazipur, Bangladesh（工程，伊斯兰技术大学，加兹ipur，孟加拉国）
Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham, Ajwad Abrar, Tareque Mohmud Chowdhury
Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.
The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses
Department of Chemistry and Applied Biosciences, ETH Zurich（应用生物科学系，苏黎世联邦理工学院）；Department of Computer Science, ETH Zurich（计算机科学系，苏黎世联邦理工学院）；Department of Health Sciences and Technology, ETH Zurich（健康科学与技术系，苏黎世联邦理工学院）
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Renmin University of China（中国人民大学）；National Yang Ming Chiao Tung University；Taipei Veterans General Hospital（台北荣民总医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Post and Telecommunications（北京邮电大学）
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
Yonsei University（延世大学）；KAIST（韩国科学技术院）；OneLineAI；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Artful Media；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Carnegie Mellon University（卡内基梅隆大学）
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
School of Computer Science and Engineering, The Hebrew University of Jerusalem（计算机科学与工程学院，特拉维夫大学）；IBM Research（IBM研究院）；The Allen Institute for AI (AI2)（人工智能研究所）
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine（上海中医药大学针灸推拿学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；China Academy of Chinese Medical Sciences（中国中医科学院）
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（信息处理国家重点实验室，计算机学院，北京大学）；StepFun；Xi’an Jiaotong University（西安交通大学）
Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals
Quality and Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；University of Marburg（马尔堡大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Deutsche Telekom（德国电信）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness
Singapore General Hospital（新加坡中央医院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Singapore National Eye Centre（新加坡国家眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务集团）；KK Women’s and Children’s Hospital（竹脚妇幼医院）；Chang Gung Memorial Hospital（长庚纪念医院）；Chang Gung University（长庚大学）；Kaohsiung Chang Gung Memorial Hospital（高雄长庚纪念医院）；National Sun Yat-sen University（国立中山大学）；Brigham and Women’s Hospital（布莱根和妇女医院）；Harvard Medical School（哈佛医学院）
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Pretraining Team, xAI（预训练团队，xAI）；Hong Kong University of Science and Technology (GuangZhou)（香港科技大学（广州））
