This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once, a paradigm we call multi-problem evaluation (MPE). Unlike conventional single-problem evaluation, where a prompt presents a single problem and expects one specific answer, MPE places multiple problems together in a single prompt and assesses how well an LLM answers all these problems in a single output. Leveraging 6 classification and 12 reasoning benchmarks that already exist, we introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts. We experiment with a total of 13 LLMs from 5 model families on ZeMPE to present a comprehensive and systematic MPE. Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short. In addition, we perform in-depth further analyses and explore model-level factors that may enable multiple problem handling capabilities in LLMs. We release our corpus and code to facilitate future research.
A word association network methodology for evaluating implicit biases in LLMs compared to humans
University of Pisa, Department of Computer Science(比萨大学计算机科学系);National Research Council of Italy, Institute of Information Science and Technologies(意大利国家研究委员会信息科学与技术研究所);University of Trento, Department of Psychology and Cognitive Science(特伦托大学心理学与认知科学系)
EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models
Fudan University(复旦大学);School of Computer Science, Fudan University(复旦大学计算机科学学院);Beijing Yuandian Technology(北京原点科技);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院)
InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles
Nankai University(南开大学);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Johns Hopkins University(约翰霍普金斯大学);Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学)
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University(国王 Khalid 大学计算机科学系);Department of Computer Science and Artificial Intelligence, Umm Al-Qura University(乌姆·阿勒·库拉大学计算机科学与人工智能系);Imam Abdulrahman Bin Faisal University(伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学)
Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs
Saudi Data & AI Authority (SDAIA)(沙特数据与人工智能局);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Princess Nourah Bint Abdulrahman University (PNU)(普罗瑟·努拉·本·阿卜杜勒拉赫曼大学)
YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese
Kobe University(神户大学);National Institute of Advanced Industrial Science and Technology (AIST)(国立研究开发法人产业技术综合研究所(AIST));The University of Tokyo(东京大学);RIKEN(理化学研究所);Tohoku University(东北大学)
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Tongji University(同济大学);AIR, Tsinghua University(清华大学人工智能研究院);BAAI(百度人工智能研究院)
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments
South China University of Technology(华南理工大学);The Hong Kong University of Science and Technology(香港科技大学);Salesforce AI Research(Salesforce AI研究院);Tsinghua University(清华大学)