Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.
Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Beijing Institute of Technology(北京理工大学海量语言信息处理与云计算应用北京工程研究中心)
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy
XiaoMi AI Lab(小米AI实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Electronic Science and Technology of China(电子科技大学)
Humans and LLMs Diverge on Probabilistic Inferences
McGill University(麦吉尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);University of Vienna(维也纳大学);FNRS – UCLouvain(佛罗伦萨-鲁汶大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers
National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);Artificial Intelligence Research Institute(人工智能研究院);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);University of California, Santa Cruz(加州大学圣克鲁兹分校);University of Texas, Dallas(德克萨斯大学达拉斯分校);University of Minnesota(明尼苏达大学)
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Huazhong University of Science and Technology(华中科技大学);The University of Hong Kong(香港大学);ByteDance Seed(字节跳动种子);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身人工智能重点实验室)
Order Doesn't Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentation
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(上海数据科学关键实验室,计算机科学学院,复旦大学);School of Data Science, Fudan University(数据科学学院,复旦大学);Ant Group(蚂蚁集团)
Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
Tsinghua University(清华大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);The University of Tokyo(东京大学);Peking University(北京大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);HKUST(香港科技大学)
FOLIO: Natural Language Reasoning with First-Order Logic
Yale University(耶鲁大学);Harvard University(哈佛大学);NVIDIA(英伟达);Iowa City West High School(爱荷华城西部高中);University of Washington(华盛顿大学);University of Hong Kong(香港大学);Penn State University(宾夕法尼亚州立大学);Salesforce Research(Salesforce研究院);Meta AI
Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning
The Laboratory for Human-Centered AI, Institute for Ethics in AI, Faculty of Philosophy, University of Oxford, UK(以人为中心的人工智能实验室,人工智能伦理研究所,哲学学院,牛津大学,英国);Oxford Internet Institute, University of Oxford, UK(牛津互联网研究所,牛津大学,英国);School of Computer Science, University of Birmingham, UK(计算机科学学院,伯明翰大学,英国)
Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction
Peking University Law School(北京大学法学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);PKU-WUHAN Institute for Artificial Intelligence(北大-武汉人工智能研究院)