ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems
Noel Thomas
Noel Thomas
备注:
7 pages, 0 figures, Accepted to AAAI-26 Bridge Program: Logical and Symbolic Reasoning in Language Models (camera-ready). v2: corrected bibliographic errors in the references
发表信息:
AAAI 2026 Bridge Program on Logical and Symbolic Reasoning in Language Models, Singapore, Jan 2026
Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);University of Toronto(多伦多大学);Birkbeck, University of London(伦敦大学伯贝克学院)
Constraints-Guided Diffusion Reasoner for Neuro-Symbolic Learning
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院);INFTECH;University of Waterloo(滑铁卢大学)
Neural-Symbolic Models for Logical Queries on Knowledge Graphs
Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);HEC Montréal(蒙特利尔高等商学院);CIFAR AI Chair(加拿大高等研究院人工智能讲席)
LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系);School of Computer Science and Technology, Tiangong University(天津大学计算机科学与技术学院);Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China(中国教育部长智网络与网络安全重点实验室);Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China(陕西省大数据知识工程重点实验室)
Neural-Symbolic Logic Query Answering in Non-Euclidean Space
Institute for Clarity in Documentation(清晰文档研究所);Inria Paris-Rocquencourt(巴黎-罗克琴堡研究所);Rajiv Gandhi University(拉朱·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒尔研究实验室)
Concise and Logically Consistent Conformal Sets for Neuro-Symbolic Concept-Based Models
Department of Information Engineering and Computer Science, University of Trento, Italy(特伦托大学信息工程与计算机科学系);CIMeC, University of Trento, Rovereto, Italy(特伦托大学罗韦雷托CIMeC)
Neuro-Symbolic Learning for Long-Horizon Task Planning Under Complex Logical Constraints
Spatial AI & Robotics (SAIR) Lab, University at Buffalo, NY 14260(空间人工智能与机器人实验室,布法罗大学,纽约州,14260);Robotics Institute, Carnegie Mellon University, PA 15213(机器人研究所,卡内基梅隆大学,宾夕法尼亚州,15213)
Extending Complex Logical Queries on Uncertain Knowledge Graphs
Department of Mathematical Sciences, Tsinghua University, Beijing, China(清华大学数学科学系);CSE, HKUST, HKSAR, China(香港科技大学计算机科学与工程系);Department of Computer Science, Princeton University, Princeton, United States(普林斯顿大学计算机科学系)
ANDRE: An Attention-based Neuro-symbolic Differentiable Rule Extractor for Inductive Logic Programming
Dept. of Mechanical and Aerospace Engineering, George Washington University, USA(机械与航空航天工程系,乔治华盛顿大学);Dept. of Mechanical Engineering Sciences, University of Surrey, UK(机械工程科学系,萨里大学)
Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction
Technical University of Darmstadt(达姆施塔特工业大学);National Research Center for Applied Cybersecurity(国家应用网络安全研究中心);Eindhoven University of Technology(埃因霍温理工大学);Hessian Center for AI (hessian.AI)(黑森人工智能中心);German Research Center for AI (DFKI)(德国人工智能研究中心)
Faculty of Engineering, Free University of Bozen-Bolzano(博洛尼亚-博尔扎诺自由大学工程学院);Department of Computer, Control and Management Engineering, Sapienza, Università di Roma(罗马大学计算机、控制与管理工程系)
Neural Scalable Symbolic Search Framework for Complex Logical Queries with Multiple Free Variables
Department of Mathematical Sciences, Tsinghua University(清华大学数学科学系);Squarepoint Capital(Squarepoint资本);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科学与技术大学计算机科学与工程系);Department of Computer Sciences, University of Rochester(罗切斯特大学计算机科学系)
Systematic Abductive Reasoning via Diverse Relation Representations in Vector-symbolic Architecture
School of Life Science and Technology, University of Electronic Science and Technology of China (UESTC)(电子科技大学生命科学与技术学院);Brain Health Institute, National Center for Mental Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine(上海交通大学医学院附属精神卫生中心国家精神障碍中心脑健康研究所);State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University(北京师范大学认知神经科学与学习国家重点实验室);School of Systems Science, Beijing Normal University(北京师范大学系统科学学院)