P-JEPA: 通过联合嵌入预测架构进行程序性视频表示学习
P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture
AI总结：
提出P-JEPA，一种与骨干网络无关的方法，通过将长视频映射到密集帧对齐动作空间并预测池化掩码潜在向量，实现对超过30分钟程序性视频的长时理解，在EgoExo4D等数据集上达到细粒度动作分类SOTA，参数比基于LLM的方法少一个数量级。
摘要：
具身AI平台的日益成熟推动了对程序性视频表示学习的兴趣，以支持复杂多步任务的智能辅助系统。利用大规模潜在预测训练，视频基础模型捕捉视频动态，从而支持活动理解、时空定位和预测控制等下游任务。然而，程序性视频包含具有长程依赖的动作，由于自注意力的二次复杂度，这些模型无法支持。例如，不同的动作可能在视觉上相似，尽管出现在程序的不同阶段，如打开和关闭炉子。本文提出一种与骨干网络无关的方法，通过将问题简化为密集的帧对齐动作空间并预测池化的掩码潜在向量，来学习长视频表示。该方法使我们的程序性联合嵌入预测架构（P-JEPA）能够处理超过30分钟的视频，从而有效理解程序步骤的长时结构。我们使用VJEPA2.1、TSM和I3D在EgoExo4D、EgoProceL和Assembly101数据集上提取特征进行评估，发现P-JEPA在线性可分性、流式推理和时间动作分割性能上持续改进，在EgoExo4D细粒度动作分类上达到最先进结果，同时参数比基于LLM的方法少一个数量级，并实现实时运行。
英文摘要：
The increasing maturity of embodied AI platforms has driven a growing interest in procedural video representation learning to support intelligent assistance systems for complex, multi-step tasks. Leveraging large-scale latent predictive training, video foundation models capture video dynamics, enabling downstream tasks such as activity understanding, spatiotemporal localization, and predictive control. However, procedural videos include actions with long-range dependencies that these models do not support, due to the quadratic complexity of self-attention. Distinct actions, for example, may be visually similar despite appearing at different points in the procedure, such as turning the stove on versus off. Here, we propose a backbone-agnostic approach that learns long-duration video representations by reducing the problem to a dense, frame-aligned action space and predicting pooled masked latent vectors. This approach allows our Procedural Joint Embedding Predictive Architecture (P-JEPA) to ingest videos over 30 minutes long, enabling effective long-form understanding of procedural steps. We evaluate P-JEPA using features extracted with VJEPA2.1, TSM, and I3D over the EgoExo4D, EgoProceL, and Assembly101 datasets, finding that it consistently improves linear separability, streaming inference, and temporal action segmentation performance, achieving state-of-the-art results on EgoExo4D fine-grained action classification while using an order of magnitude fewer parameters than LLM-based methods and running in real time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
