While large language models (LLMs) have been thoroughly evaluated for deductive and inductive reasoning, their proficiency in holistic rule learning in interactive environments remains less explored. We introduce RULEARN, a novel benchmark to assess the rule-learning abilities of LLM agents in interactive settings. In RULEARN, agents strategically interact with simulated environments to gather observations, discern patterns, and solve complex problems. To enhance the rule-learning capabilities for LLM agents, we propose IDEA, a novel reasoning framework that integrates the process of Induction, Deduction, and Abduction. The IDEA agent generates initial hypotheses from limited observations through abduction, devises plans to validate these hypotheses or leverages them to solve problems via deduction, and refines previous hypotheses through induction, dynamically establishing and applying rules that mimic human rule-learning behaviors. Our evaluation of the IDEA framework, which involves five representative LLMs, demonstrates significant improvements over the baseline. Furthermore, our study with human participants reveals notable discrepancies in rule-learning behaviors between humans and LLMs. We believe our benchmark will serve as a valuable and challenging resource, and IDEA will provide crucial insights for the development of LLM agents capable of human-like rule learning in real-world scenarios. Our code and data is publicly available.
ItD: Large Language Models Can Teach Themselves Induction through Deduction
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc(小红书公司);Texas A&M University(德克萨斯农工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
School of Computer Science & Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Institute for AI, Peking University(北京大学人工智能研究院);Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China(北京理工大学长三角研究院(嘉兴));National Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能全国重点实验室)
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Idiap Research Institute(Idiap研究 institute);EPFL(瑞士联邦理工学院);School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);University of Manchester(曼彻斯特大学);National Biomarker Centre, CRUK Manchester Institute(国家生物标志物中心,CRUK曼彻斯特研究所)
Compliance vs. Sensibility: On the Reasoning Controllability in Large Language Models
School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);School of EECS, Queen Mary University of London(伦敦女王学院电子工程与计算机科学学院);The Alan Turing Institute(艾伦·图灵研究所)
Medical Reasoning with Large Language Models: A Survey and MR-Bench
University of Science and Technology of China(中国科学技术大学);The First Affiliated Hospital of USTC(中国科学技术大学附属第一医院);National University of Singapore(新加坡国立大学)
LLM Agents as Social Scientists: A Human-AI Collaborative Platform for Social Science Automation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室)
The Reasoning Trap -- Logical Reasoning as a Mechanistic Pathway to Situational Awareness
MARS 4.0 Fellowship, Cambridge AI Safety Hub(CAISH), University of Cambridge(MARS 4.0 Fellow,剑桥人工智能安全中心(CAISH),剑桥大学);AWS Generative AI Innovation Center, Amazon Web Services, USA(亚马逊生成AI创新中心,亚马逊网络服务,美国);Google, USA(谷歌,美国);Stanford University(斯坦福大学);Northeastern University, Seattle, WA, USA(东北大学,西雅图,华盛顿州,美国)
Shuttle Between the Instructions and the Parameters of Large Language Models
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(认知与决策智能复杂系统实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Xiaohongshu Inc(小红书公司);Kuaishou Technology(快手技术);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)