arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.01982cs.AIcs.ITmath.IT
收藏

ChaosBench-Logic:逻辑和符号推理在混沌动力系统上的基准

ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

Noel Thomas

Noel Thomas

AI总结:

ChaosBench-Logic通过30种混沌动力系统评估LLM逻辑和符号推理能力,揭示其在复杂推理任务中的局限性。

摘要:

大型语言模型(LLMs)在自然语言任务中表现出色,但在需要精确逻辑和符号推理的领域中却显得脆弱。混沌动力系统提供了特别具有挑战性的测试,因为混沌是确定性的,但往往被误认为随机或复杂。我们介绍了ChaosBench-Logic,这是一个评估LLM推理能力的基准,通过30种多样的动力系统使用统一的一阶逻辑(FOL)本体进行评估。每个系统都标注了11个语义谓词的真值分配,并在七个推理类别中生成了621个问题,包括多跳推理、跨系统类比、反事实推理、偏见探测和多轮对话。我们定义了逻辑准确性、推理一致性、对话连贯性和矛盾等指标,并发布了开源评估流程。初步实验表明,前沿LLMs如GPT-4、Claude 3.5 Sonnet、Gemini 2.5 Flash和开源的LLaMA-3 70B在单个项目准确性上达到91-94%,但仍然在组合项目上得分为0%,且表现出脆弱的全局连贯性。对话层面的准确性范围从53.1%(GPT-4 CoT)到75.5%(LLaMA-3零样本)。ChaosBench-Logic为诊断此类失败提供了严格测试平台,并为开发改进LLM科学推理的神经符号方法奠定了基础。

英文摘要:

Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑