arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01171cs.AIcs.CLcs.LO
收藏

ASP-Bench: 从自然语言到逻辑程序

ASP-Bench: From Natural Language to Logic Programs

Algorithms and Complexity Group TU Wien(算法与复杂性组维也纳技术大学)

Stefan Szeider

AI总结:

ASP-Bench通过自然语言到逻辑程序的转换基准,评估了基于ReAct框架的代理方法,揭示了建模难度的多维因素。

摘要:

将自然语言规范自动转换为逻辑程序是一项具有挑战性的任务,影响着神经符号工程。我们提出了ASP-Bench,包含128个自然语言问题实例、64个基础问题(易变体和难变体)。它评估将自然语言问题转换为答案集程序(ASPs)的系统,ASPs是逻辑编程的主要形式之一。它系统地覆盖了ASPs的特征,包括选择规则、聚合和优化。每个问题都包含参考验证器,检查解决方案是否满足问题规范。我们沿七个大致独立的推理方面(优化、时间推理、默认逻辑、资源分配、递归、空间推理和量化复杂性)对问题进行了特征化,提供了对建模难度的多维视角。我们使用基于ReAct(推理和行动)框架的代理方法测试了该基准,实现了完全饱和,证明了反馈驱动的迭代细化与求解器反馈提供了一种可靠且稳健的方法来在ASPs中建模自然语言。我们对多个代理运行的分析使我们能够获得影响问题建模难度的因素的见解。

英文摘要:

Automating the translation of natural-language specifications into logic programs is a challenging task that affects neurosymbolic engineering. We present ASP-Bench, a benchmark comprising 128 natural language problem instances, 64 base problems with easy and hard variants. It evaluates systems that translate natural-language problems into Answer Set Programs (ASPs), a prominent form of logic programming. It provides systematic coverage of ASP features, including choice rules, aggregates, and optimization. Each problem includes reference validators that check whether solutions satisfy the problem specification. We characterize problems along seven largely independent reasoning aspects (optimization, temporal reasoning, default logic, resource allocation, recursion, spatial reasoning, and quantitative complexity), providing a multidimensional view of modeling difficulty. We test the benchmark using an agentic approach based on the ReAct (Reason and Act) framework, which achieves full saturation, demonstrating that feedback-driven iterative refinement with solver feedback provides a reliable and robust approach for modeling natural language in ASP. Our analysis across multiple agent runs enables us to gain insights into what determines a problem's modeling hardness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑