Multiple choice questions (MCQs) are commonly used to evaluate the capabilities of large language models (LLMs). One common way to evaluate the model response is to rank the candidate answers based on the log probability of the first token prediction. An alternative way is to examine the text output. Prior work has shown that first token probabilities lack robustness to changes in MCQ phrasing, and that first token probabilities do not match text answers for instruction-tuned models. Therefore, in this paper, we investigate the robustness of text answers. We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers. The difference in robustness increases as the mismatch rate becomes greater. As the mismatch reaches over 50\%, the text answer is more robust to option order changes than the debiased first token probabilities using state-of-the-art debiasing methods such as PriDe. Our findings provide further evidence for the benefits of text answer evaluation over first token probability evaluation.
RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format
Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理国家重点实验室(上海))
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University(昌原大学);Korea University(韩国大学);Seoul National University(首尔国立大学);Korea Institute for Curriculum and Evaluation(韩国课程与评价研究院);Upstage;Indiana University Bloomington(印第安纳大学布卢明顿分校);Nanyang Technological University(南洋理工大学)
Data Selection for Multi-turn Dialogue Instruction Tuning
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);School of Computer Science, Peking University(北京大学计算机学院);PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications(北京大学-湖北CMCC大模型工业应用联合实验室)
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);School of Computer Science, Peking University(北京大学计算机学院);PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications(北京大学-CMCC(湖北)大语言模型工业应用联合实验室);School of Health Sciences and Biomedical Engineering, Hebei University of Technology(河北工业大学健康科学与生物医学工程学院)
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Kyoto University(京都大学);Tencent AI Lab(腾讯AI实验室);Tomorrow Advancing Life(明天生命科技)
Patches of Nonlinearity: Instruction Vectors in Large Language Models
Ubiquitous Knowledge Processing Lab (UKP Lab)(通用知识处理实验室);Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(计算机科学系,达姆施塔特技术大学和应用网络安全国家研究中心ATHENE,德国)
Reasoning Up the Instruction Ladder for Controllable Language Models
Department of Computer Science and Engineering, The Ohio State University(俄亥俄州立大学计算机科学与工程系);Microsoft Research(微软研究院);Allen Institute for AI(艾伦人工智能研究所)
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
School of Artificial Intelligence and Computer Science, Jilin University of Finance and Economics(吉林财经大学人工智能与计算机科学学院);Jilin Province Key Laboratory of Fintech, Jilin University of Finance and Economics(吉林省金融科技重点实验室,吉林财经大学);School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology(奥克兰理工大学工程、计算机与数学科学学院)
ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy
National Key Laboratory for Multimedia Information Processing, Peking University(国家多媒体信息处理重点实验室,北京大学);School of Software and Microelectronics, Peking University(软件与微电子学院,北京大学);School of Computer Science, Peking University(计算机学院,北京大学)
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Information Sciences Institute, Viterbi School of Engineering, University of Southern California(南加州大学维特比工程学院信息科学研究所);Department of Computer Science, Faculty of Science, Vrije Universiteit Amsterdam(阿姆斯特丹自由大学理学院计算机科学系);Tencent AI Lab(腾讯AI实验室)
How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction
Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海人工智能教育研究院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学人工智能教育实验室)
Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs
Arizona State University(亚利桑那州立大学);Morgan Stanley(摩根士丹利);UC Davis(加州大学戴维斯分校);Washington University in St. Louis(圣路易斯华盛顿大学);Rice University(莱斯大学);Florida State University(佛罗里达州立大学);University of Oxford(牛津大学)
Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
Hasso Plattner Institute for Digital Engineering, University of Potsdam(波茨坦大学哈索·普拉特纳数字工程研究所);Qualcomm AI Research(高通人工智能研究);University of Amsterdam(阿姆斯特丹大学)
Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心);ETH Zurich(苏黎世联邦理工学院);University Hospital Tübingen(图宾根大学医院);University of Tübingen(图宾根大学);ELLIS Institute Tübingen(图宾根ELLIS研究所)
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY(CUNY研究生中心);Brooklyn College, CUNY(CUNY布鲁克林学院);Borough of Manhattan Community College, CUNY(CUNY曼哈顿社区学院);The City College of New York, CUNY(CUNY纽约城市学院)