通过基于状态的层次解折叠学习过程感知的视频表示
Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding
AI总结：
本文提出TSS框架，通过基于状态的层次解折叠学习过程感知的视频表示，提升任务识别和步骤预测性能。
摘要：
学习过程感知的视频表示是构建能够推理和执行复杂任务的代理的关键步骤。现有方法通常通过在任务和步骤层面对视觉内容与文本描述进行对齐，将过程语义注入视频表示中。然而，由于其高度抽象性，'任务'和'步骤'描述无法与视觉数据中具体的、可观察的细节形成稳健的对齐。为此，我们引入'状态'，即对象配置的文本快照，作为一种视觉上 grounded 的语义层，将抽象过程锚定在模型实际能看到的内容上。我们在这个洞察的基础上，提出了一个新的任务-步骤-状态（TSS）框架，其中任务通过步骤驱动在可观测状态之间的转换。为了强制这种结构，我们提出了一个渐进式预训练策略，通过解折叠TSS层次结构，迫使模型在状态上建立表示，同时将它们与步骤和高层任务关联起来。在COIN和CrossTask数据集上的大量实验表明，我们的方法在多个下游任务上优于基线模型，包括任务识别、步骤识别和下一步预测。消融研究显示，引入状态监督是跨所有任务性能提升的关键驱动因素。此外，我们的渐进式预训练策略比标准联合训练更有效，因为它更好地强制了预期的层次结构。
英文摘要：
Learning procedural-aware video representations is a key step towards building agents that can reason about and execute complex tasks. Existing methods typically address this problem by aligning visual content with textual descriptions at the task and step levels to inject procedural semantics into video representations. However, due to their high level of abstraction, 'task' and 'step' descriptions fail to form a robust alignment with the concrete, observable details in visual data. To address this, we introduce 'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to what a model can actually see. We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states. To enforce this structure, we propose a progressive pre-training strategy that unfolds the TSS hierarchy, forcing the model to ground representations in states while associating them with steps and high-level tasks. Extensive experiments on the COIN and CrossTask datasets show that our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Ablation studies show that introducing state supervision is a key driver of performance gains across all tasks. Additionally, our progressive pretraining strategy proves more effective than standard joint training, as it better enforces the intended hierarchical structure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
