神经符号推理器在一阶逻辑问题上的鲁棒性
Robustness of Neurosymbolic Reasoners on First-Order Logic Problems
AI总结：
本文研究神经符号方法在一阶逻辑反事实变体上的鲁棒性，发现其比纯神经方法更鲁棒但整体性能较差，并提出结合思维链的NSCoT方法，虽有所改进但仍不及标准CoT。
摘要：
自然语言处理（NLP）的最新趋势旨在提升大型语言模型（LLM）的推理能力，重点关注对任务变化的泛化能力和鲁棒性。反事实任务变体对原本有效的一阶逻辑（FOL）问题实例引入最小但语义上有意义的变化，例如更改单个谓词或交换常量的角色，以探测推理系统在扰动下是否能够保持逻辑一致性。先前的研究表明，LLM在反事实变体上变得脆弱，这表明它们往往依赖虚假的表面模式来生成响应。在这项工作中，我们探索了一种将LLM与符号逻辑求解器相结合的神经符号（NS）方法是否能够缓解这一问题。在多种规模的LLM上进行的实验表明，NS方法更加鲁棒，但整体表现不如纯神经方法。随后，我们提出了NSCoT，该方法将NS方法与思维链（CoT）提示相结合，并证明虽然NSCoT能够提升性能，但仍然落后于标准CoT。我们的分析为未来工作开辟了研究方向。
英文摘要：
Recent trends in NLP aim to improve reasoning capabilities in Large Language Models (LLMs), with key focus on generalization and robustness to variations in tasks. Counterfactual task variants introduce minimal but semantically meaningful changes to otherwise valid first-order logic (FOL) problem instances altering a single predicate or swapping roles of constants to probe whether a reasoning system can maintain logical consistency under perturbation. Previous studies showed that LLMs becomes brittle on counterfactual variations, suggesting that they often rely on spurious surface patterns to generate responses. In this work, we explore if a neurosymbolic (NS) approach that integrates an LLM and a symbolic logical solver could mitigate this problem. Experiments across LLMs of varying sizes show that NS methods are more robust but perform worse overall that purely neural methods. We then propose NSCoT that combines an NS method and Chain-of-Thought (CoT) prompting and demonstrate that while it improves performance, NSCoT still lags behind standard CoT. Our analysis opens research directions for future work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
