瞬间动作预见：多模态线索能替代视频到何种程度？
Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?
AI总结：
AAG通过结合单帧RGB特征与深度线索及先前动作信息，实现了多模态单帧动作预见，能与视频聚合基线和先进方法在教学活动数据集上竞争。
摘要：
在动作理解研究中，提前预测动作是一项核心挑战。虽然传统方法依赖于从视频中提取和聚合时间信息，但作为人类，我们往往可以通过观察场景中的单个时刻来预测即将发生的行为，前提是有足够的上下文。模型能否实现这种能力？简短的回答是肯定的，尽管其有效性取决于任务的复杂性。在本工作中，我们探讨视频聚合可以被其他模态替代到何种程度。为此，基于最近的视觉特征提取和基于语言的推理进展，我们引入了AAG，一种瞬间动作预见方法。AAG结合单帧图像的RGB特征与深度线索以增强空间推理，并结合先前的动作信息以提供长期上下文。该上下文可通过Vision-Language Models生成的文本摘要或通过单帧动作识别器生成的预测来获得。我们的结果表明，使用AAG进行多模态单帧动作预见在三个教学活动数据集（IKEA-ASM、Meccano和Assembly101）上可以与时间聚合视频基线和最先进的方法相竞争。
英文摘要：
Anticipating actions before they occur is a core challenge in action understanding research. While conventional methods rely on extracting and aggregating temporal information from videos, as humans we can often predict upcoming actions by observing a single moment from a scene, when given sufficient context. Can a model achieve this competence? The short answer is yes, although its effectiveness depends on the complexity of the task. In this work, we investigate to what extent video aggregation can be replaced with alternative modalities. To this end, based on recent advances in visual feature extraction and language-based reasoning, we introduce AAG, a method for Action Anticipation at a Glimpse. AAG combines RGB features with depth cues from a single frame for enhanced spatial reasoning, and incorporates prior action information to provide long-term context. This context is obtained either through textual summaries from Vision-Language Models, or from predictions generated by a single-frame action recognizer. Our results demonstrate that multimodal single-frame action anticipation using AAG can perform competitively compared to both temporally aggregated video baselines and state-of-the-art methods across three instructional activity datasets: IKEA-ASM, Meccano, and Assembly101.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
