PARADISE:利用程序性警告与提示数据集评估语言模型的隐式规划能力
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset
AI总结:
本文提出PARADISE数据集,通过基于wikiHow的警告与提示问答任务评估语言模型从目标推断隐式规划知识的能力,发现小型微调模型在多数场景优于大模型,但所有模型仍不及人类。
摘要:
近期,社区中关于大型语言模型是否能够进行规划或执行计划的兴趣日益增长。然而,大多数先前研究使用LLM为缺乏语言复杂性和领域多样性的简化场景生成高层计划,限制了对它们规划能力的分析。这些设置约束了评估方法(例如预定义动作空间)、架构选择(例如仅生成式模型),并忽视了对于现实分析至关重要的语言细微差别。为解决这一问题,我们提出了PARADISE,这是一个基于wikiHow实用程序性文本、采用问答形式的溯因推理任务。该任务包含与目标直接相关的警告和提示推理任务,排除中间步骤,旨在测试模型仅根据给定目标推断计划隐式知识的能力。我们的实验利用微调语言模型和零样本提示,揭示了任务特定的小型模型在大多数场景下优于大型语言模型。尽管取得了进展,所有模型仍不及人类表现。值得注意的是,我们的分析发现了有趣的见解,例如模型在关键词被删除时行为的变化、BERT系列和GPT-4在物理目标和抽象目标上的困难,以及所提出的任务为其他未见过的程序性任务提供了有价值的先验知识。PARADISE数据集及相关资源已公开可用,供进一步研究探索:https://github.com/GGLAB-KU/paradise。
英文摘要:
Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。