Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. To investigate this, we propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic, comprising both primitive and compositional rules across five domains. Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs' logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns. We further distill these rules into a smaller-scale inference engine for flexible rule generation and enhancing downstream reasoning. Through a multi-judger evaluation, our inference engine proves effective in generating accurate, complex and abstract conclusions and premises, and improve various commonsense reasoning tasks. Overall, our work sheds light on LLMs' limitations in grasping inferential rule and suggests ways to enhance their logical reasoning abilities~\footnote{Code and data are available at \url{https://github.com/SiyuanWangw/ULogic}.}.
HiMed: Incentivizing Hindi Reasoning in Medical LLMs
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Indian Institute of Technology (Banaras Hindu University) Varanasi(印度理工学院(班加罗尔 Hindu 大学)瓦拉纳西分校);Tongji University(同济大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Shenzhen Loop Area Institute(深圳河套学院);The Hong Kong University of Science and Technology(香港科技大学);Halmstad University(哈尔姆斯塔德大学)
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Bar-Ilan University(巴伊兰大学);University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(Vector人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)
Reasoning that Travels: Dissecting How Chain-of-Thought Transfers Across Models
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(MaiNLP,信息与语言处理中心,慕尼黑大学,德国);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心,德国)
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering(奥林工程学院);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Oxford(牛津大学);University College London(伦敦大学学院);ShanghaiTech University(上海科技大学);Columbia University(哥伦比亚大学);University of Manchester(曼彻斯特大学)
MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks
Program in Computational Biology & Biomedical Informatics, Yale University(计算生物学与生物医学信息学项目,耶鲁大学);Department of Biomedical Informatics & Data Science, Yale University(生物医学信息学与数据科学系,耶鲁大学);Department of Computer Science, Yale University(计算机科学系,耶鲁大学);Department of Biosystems Science and Engineering, ETH Zurich(生物系统科学与工程系,苏黎世联邦理工学院);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学);Department of Bioinformatics, UT Southwestern Medical Center(生物信息学系,德克萨斯西南医学中心)
Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning
Dept. of Electrical and Computer Engineering, Texas A&M University(电气与计算机工程系,德克萨斯A&M大学);College of Science and Engineering, Hamad Bin Khalifa University(科学与工程学院,哈马德·本·卡西姆大学);Dept. of Electrical and Computer Engineering, Texas A&M University at Qatar(电气与计算机工程系,德克萨斯A&M大学(卡塔尔));Dept. of Prosthetics and Orthotics, Ankara University(假肢与矫形系,安卡拉大学)
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Department of Mechanical Engineering(机械工程系);Massachusetts Institute of Technology(麻省理工学院);Department of Civil and Environmental Engineering(土木与环境工程系);Schwarzman College of Computing(施瓦茨曼计算学院)
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);Seoul National University(首尔大学)
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能研究所);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Ant Group(蚂蚁集团)
Cognition Chain for Explainable Psychological Stress Detection on Social Media
University of Oxford(牛津大学);Tsinghua University(清华大学);Beijing Normal University(北京师范大学);Wuhan University(武汉大学);Oxford Suzhou Centre for Advanced Research(牛津苏州高等研究院)
StressPrompt: Does Stress Impact Large Language Models and Human Performance Similarly?
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Center for Long-term Artificial Intelligence(长期人工智能中心);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院)
Modeling Grammatical Hypothesis Testing in Young Learners: A Sequence-Based Learning Analytics Study of Morphosyntactic Reasoning in an Interactive Game