arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.00092cs.CL
收藏

PROC2PDDL:从文本获得开放域规划表示

PROC2PDDL: Open-Domain Planning Representations from Texts

University of Pennsylvania(宾夕法尼亚大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所)

Tianyi Zhang, Li Zhang, Zhaoyi Hou, Ziyu Wang, Yuling Gu, Peter Clark, Chris Callison-Burch, Niket Tandon

AI总结:

本文提出首个开放域过程性文本到专家标注 PDDL 表示的 Proc2PDDL 数据集,并评测主流 LLM,发现其动作前提与效果建模表现很差,揭示了程序生成和事件推理缺陷。

摘要:

在基于文本的环境中进行规划,仍然是 AI 系统面临的一项重大挑战。近期方法已使用语言模型预测规划领域定义(例如 PDDL),但这些方法只在封闭域模拟环境中接受过评估。为解决这一问题,我们提出 Proc2PDDL,这是首个包含开放域过程性文本并配有专家标注 PDDL 表示的数据集。利用该数据集,我们评估了最先进模型在定义动作的前提条件和效果方面的表现。我们表明 Proc2PDDL 极具挑战性:GPT-3.5 的成功率接近 0%,而 GPT-4 约为 35%。我们的分析揭示了句法和语义两类错误,表明语言模型在生成特定领域程序以及对事件进行推理两方面均存在不足。我们希望这一分析和数据集有助于未来在整合语言模型与形式化规划各自优势方面取得进展。

英文摘要:

Planning in a text-based environment continues to be a major challenge for AI systems. Recent approaches have used language models to predict a planning domain definition (e.g., PDDL) but have only been evaluated in closed-domain simulated environments. To address this, we present Proc2PDDL , the first dataset containing open-domain procedural texts paired with expert-annotated PDDL representations. Using this dataset, we evaluate state-of-the-art models on defining the preconditions and effects of actions. We show that Proc2PDDL is highly challenging, with GPT-3.5's success rate close to 0% and GPT-4's around 35%. Our analysis shows both syntactic and semantic errors, indicating LMs' deficiency in both generating domain-specific prgorams and reasoning about events. We hope this analysis and dataset helps future progress towards integrating the best of LMs and formal planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑