arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15823cs.CL
收藏

CaT-BENCH:基准评测语言模型对计划中因果与时间依赖关系的理解

CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans

Stony Brook University(石溪大学);University of Texas, Austin(德克萨斯大学奥斯汀分校);US Naval Academy(美国海军学院)

Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, Raymond Mooney

AI总结:

本文提出 CaT-Bench,通过菜谱步骤顺序预测评测 LLM 对因果与时间依赖的理解,发现 SOTA 模型 F1 最高仅 0.73,且推理一致性和人类认可度不足。

摘要:

理解 LLM 对自然语言计划(例如说明文本和菜谱)进行推理的能力,对于在决策系统中可靠使用它们至关重要。计划的一个基本方面是其步骤需要被执行的时间顺序,这反映了步骤之间潜在的因果依赖。我们提出 CaT-Bench,这是一个包含步骤顺序预测问题的基准,用于测试在烹饪菜谱计划中,某个步骤是否必须必然发生在另一个步骤之前或之后。我们利用该基准评估前沿 LLM 对因果和时间依赖的理解程度。我们发现,SOTA LLM 的表现并不理想(最佳零样本结果的 F1 仅为 0.59),并且偏向于更频繁地预测存在依赖,这或许是因为它们依赖步骤的时间顺序作为启发式方法。虽然提示模型给出解释以及使用少样本示例可以提升性能,但最佳 F1 结果也仅为 0.73。此外,结合答案正确性对解释进行人工评估表明,平均而言,人类并不认同模型的推理。令人惊讶的是,我们还发现,先回答后解释比普通的思维链提示带来更好的性能,而且 LLM 对相同步骤对的不同问题给出的答案并不一致。总体而言,结果表明 LLM 检测步骤之间依赖关系的能力仍有显著提升空间。

英文摘要:

Understanding the abilities of LLMs to reason about natural language plans, such as instructional text and recipes, is critical to reliably using them in decision-making systems. A fundamental aspect of plans is the temporal order in which their steps needs to be executed, which reflects the underlying causal dependencies between them. We introduce CaT-Bench, a benchmark of Step Order Prediction questions, which test whether a step must necessarily occur before or after another in cooking recipe plans. We use this to evaluate how well frontier LLMs understand causal and temporal dependencies. We find that SOTA LLMs are underwhelming (best zero-shot is only 0.59 in F1), and are biased towards predicting dependence more often, perhaps relying on temporal order of steps as a heuristic. While prompting for explanations and using few-shot examples improve performance, the best F1 result is only 0.73. Further, human evaluation of explanations along with answer correctness show that, on average, humans do not agree with model reasoning. Surprisingly, we also find that explaining after answering leads to better performance than normal chain-of-thought prompting, and LLM answers are not consistent across questions about the same step pairs. Overall, results show that LLMs' ability to detect dependence between steps has significant room for improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑