arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.03474cs.CV
收藏

通过动作效果建模实现过程性错误检测

Procedural Mistake Detection via Action Effect Modeling

Michigan State University(密歇根州立大学)

Wenliang Guo, Yujiang Pu, Yu Kong

AI总结:

本文提出通过动作效果建模实现过程性错误检测,结合语义和视觉信息,在单类分类任务中取得最佳性能。

摘要:

过程性任务中的错误检测对于构建支持学习和任务执行的智能系统至关重要。现有方法主要分析动作是如何执行的,而忽略了其产生的结果,即动作效果。然而,许多错误并非出现在执行本身,而是在结果中表现出来,例如意外的对象状态或错误的空间排列。为解决这一差距,我们提出了动作效果建模(AEM),一个统一的框架,通过概率形式联合捕捉动作执行及其结果。AEM首先通过选择最具有信息量的效果框架来识别动作的结果,基于语义相关性和视觉质量。然后,它从视觉定位和符号场景图中提取互补的线索,并在共享的潜在空间中对齐它们,形成稳健的效果感知表示。为了检测错误,我们进一步设计了一个基于提示的检测器,该检测器结合了任务特定的提示,并将每个动作片段与其预期的执行语义对齐。我们的方法在EgoPER和CaptainCook4D基准测试中实现了最先进的性能,特别是在具有挑战性的单类分类(OCC)设置下。这些结果表明,建模执行和结果会产生更可靠的错误检测,并突显了效果感知表示在更广泛下游应用中的潜力。

英文摘要:

Mistake detection in procedural tasks is essential for building intelligent systems that support learning and task execution. Existing approaches primarily analyze how an action is performed, while overlooking what it produces, i.e., the \textbf{action effect}. Yet many errors manifest not in the execution itself but in the resulting outcome, such as an unintended object state or incorrect spatial arrangement. To address this gap, we propose Action Effect Modeling (AEM), a unified framework that jointly captures action execution and its outcomes through a probabilistic formulation. AEM first identifies the outcome of an action by selecting the most informative effect frame based on semantic relevance and visual quality. It then extracts complementary cues from visual grounding and symbolic scene graphs, aligning them in a shared latent space to form robust effect-aware representations. To detect mistakes, we further design a prompt-based detector that incorporates task-specific prompts and aligns each action segment with its intended execution semantics. Our approach achieves state-of-the-art performance on the EgoPER and CaptainCook4D benchmarks under the challenging one-class classification (OCC) setting. These results demonstrate that modeling both execution and outcome yields more reliable mistake detection, and highlight the potential of effect-aware representations to benefit a broader range of downstream applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑