arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1106.1804cs.AI
收藏

规划中基准比较的批判性评价

A Critical Assessment of Benchmark Comparison in Planning

Colorado State University(科罗拉多州立大学)

E. Dahlman, A. E. Howe

摘要:

规划研究的最新趋势导致经验比较变得司空见惯。这个领域已经开始形成一种用于这种比较的方法,出于明显的实际原因,这种方法需要在一个子集的问题上运行一个子集的规划器。在本文中,我们描述的方法,并检查八个隐含的假设的问题,规划者和指标中使用的许多这些比较。问题假设是:PR 1)如果在问题和域的抽样上执行,则通用规划器的性能不应受到惩罚/偏见,PR 2)表示中的微小语法差异不会影响性能,PR 3)问题应可由具有CNOPS能力的规划器解决,除非它们需要ADL。计划者的假设是:PL 1)最新版本的计划器是最好的,PL 2)默认参数设置近似良好的性能,PL 3)时间截止不会过度偏倚结果。度量假设是:M1)当在降级的运行时环境(例如,机器平台)和M2)计划步骤的数量区分性能。我们发现,大多数这些假设并不支持经验,特别是,规划者受到不同的影响,这些假设。最后,我们呼吁社会投入研究资源,以改善国家的做法,特别是加强现有的基准问题。

英文摘要:

Recent trends in planning research have led to empirical comparison becoming commonplace. The field has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1) the performance of a general purpose planner should not be penalized/biased if executed on a sampling of problems and domains, PR2) minor syntactic differences in representation do not affect performance, and PR3) problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1) the latest version of a planner is the best one to use, PL2) default parameter settings approximate good performance, and PL3) time cut-offs do not unduly bias outcome. The metrics assumptions are: M1) performance degrades similarly for each planner when run on degraded runtime environments (e.g., machine platform) and M2) the number of plan steps distinguishes performance. We find that most of these assumptions are not supported empirically; in particular, that planners are affected differently by these assumptions. We conclude with a call to the community to devote research resources to improving the state of the practice and especially to enhancing the available benchmark problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑