Can multiple-choice questions really be useful in detecting the abilities of LLMs?
Meetyou AI Lab（MeetYou人工智能实验室）；Southwestern University of Finance and Economics（西南财经大学）；Osaka University（大阪大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy, which we undertake in this paper by evaluating nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English. We identify a significant issue: LLMs exhibit an order sensitivity in bilingual MCQs, favoring answers located at specific positions, i.e., the first position. We further quantify the gap between MCQs and long-form generation questions (LFGQs) by comparing their direct outputs, token logits, and embeddings. Our results reveal a relatively low correlation between answers from MCQs and LFGQs for identical questions. Additionally, we propose two methods to quantify the consistency and confidence of LLMs' output, which can be generalized to other QA evaluation benchmarks. Notably, our analysis challenges the idea that the higher the consistency, the greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms of expected calibration error. Finally, the misalignment between MCQs and LFGQs is not only reflected in the evaluation performance but also in the embedding space. Our code and models can be accessed at https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs.
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；University of Macau（澳门大学）；HKUST（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；Duke Kunshan University（昆山杜克大学）
What's the Meaning of Superhuman Performance in Today's NLU?
Babelscape（巴别斯克）；Sapienza University of Rome（罗马第一大学）；University of Groningen（格罗宁根大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Charles University（查理大学）；University of Copenhagen（哥本哈根大学）；University of Melbourne（墨尔本大学）；Carnegie Mellon University（卡内基梅隆大学）；Saarland University（萨尔大学）；Jožef Stefan Institute（约泽夫·斯特凡研究所）；University of Ljubljana（卢布尔雅那大学）；Cardiff University（卡迪夫大学）；University of Zurich（苏黎世大学）；University of Edinburgh（爱丁堡大学）；University of Amsterdam（阿姆斯特丹大学）
An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Alibaba Group（阿里巴巴集团）
Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder
The Chinese University of Hong Kong（香港中文大学）；Stanley Ho Big Data Decision Analytics Research Centre, The Chinese University of Hong Kong（香港中文大学何文汇大数据决策分析研究中心）
CODEMENV: Benchmarking Large Language Models on Code Migration
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明责任人工智能与数据分析实验室）；King Abdullah University of Science and Technology（国王 Abdullah 科学与技术大学）；Peking University（北京大学）；South China University of Technology（华南理工大学）
XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
Google DeepMind（谷歌DeepMind）；Indian Institute of Technology Bombay（印度理工学院孟买分校）；International Centre for Theoretical Sciences, Tata Institute of Fundamental Research（理论科学国际中心， Tata 基础研究机构）
