arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.05972cs.CLcs.AI
收藏

LexiCon:自然语言中时间约束下规划任务的基准

LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language

Örebro University(厄勒布鲁大学);KU Leuven(鲁汶大学)

Periklis Mantenoglou, Rishi Hazra, Pedro Zuidberg Dos Martires, Luc De Raedt

AI总结:

LexiCon 是一个基于自然语言的时间约束规划基准,通过给现有环境施加时间约束并转化为语言任务,评估 LLM 的规划能力;实验表明,约束增加时,包括 GPT-5、o3 和 R1 在内的先进模型性能下降。

摘要:

由于大型语言模型(LLM)具备推理能力,它们已在用自然语言描述的规划任务上得到评估。然而,LLM 主要是在没有约束的规划领域中进行测试。为了在现实世界中部署它们,而现实世界中遵守约束(尤其是安全约束)至关重要,我们需要评估它们在受约束规划任务上的性能。我们引入了 LexiCon——一个基于自然语言(Lexi)的受约束(Con)规划基准,包含一系列环境,可用于以原则性方式评估 LLM 的规划能力。LexiCon 的核心思想是采用现有的规划环境,并对状态施加时间约束。这些受约束的问题随后被转化为自然语言,并交给 LLM 求解。LexiCon 的一个关键特性是其可扩展性。也就是说,受支持的环境集合可以通过新的(无约束)环境生成器进行扩展,而时间约束会自动为这些生成器构建。这使得 LexiCon 具有未来适应性:随着 LLM 规划能力的提升,所生成的规划问题的难度可以相应增加。我们的实验表明,最先进的 LLM(包括 GPT-5、o3 和 R1 等推理模型)的性能会随着规划任务约束程度的增加而下降。

英文摘要:

Owing to their reasoning capabilities, large language models (LLMs) have been evaluated on planning tasks described in natural language. However, LLMs have largely been tested on planning domains without constraints. In order to deploy them in real-world settings where adherence to constraints, in particular safety constraints, is critical, we need to evaluate their performance on constrained planning tasks. We introduce LexiCon -- a natural language-based (Lexi) constrained (Con) planning benchmark, consisting of a suite of environments, that can be used to evaluate the planning capabilities of LLMs in a principled fashion. The core idea behind LexiCon is to take existing planning environments and impose temporal constraints on the states. These constrained problems are then translated into natural language and given to an LLM to solve. A key feature of LexiCon is its extensibility. That is, the set of supported environments can be extended with new (unconstrained) environment generators, for which temporal constraints are constructed automatically. This renders LexiCon future-proof: the hardness of the generated planning problems can be increased as the planning capabilities of LLMs improve. Our experiments reveal that the performance of state-of-the-art LLMs, including reasoning models like GPT-5, o3, and R1, deteriorates as the degree of constrainedness of the planning tasks increases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑