从帧到时间图：基于视觉语言模型的上下文第一人称动作识别
From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
AI总结：
提出将视频转换为时间动作图，通过多阶段提示生成自然语言叙述并结构化，实现上下文学习，在EGTEA和Epic-Kitchens-100上显著提升零样本和少样本动作识别性能。
摘要：
第一人称视频中的动作推理需要捕捉手-物交互的细粒度过渡，而通用视觉语言模型（VLM）在直接处理原始像素时往往难以胜任。我们提出通过将视频转换为时间动作图，将视觉感知与符号推理解耦。在多阶段提示流程中，我们首先在短时间窗口上生成密集的自然语言叙述作为语义瓶颈，然后将其形式化为结构化的开放词汇图表示。在EGTEA和Epic-Kitchens-100数据集上，符号表示实现了高效的上下文学习：少样本图演示相比零样本帧和图推理均带来显著的准确率提升。即使在零样本设置下，尽管潜在的预训练污染可能有利于基于像素的推理，但基于图的推理仍能与像素推理保持竞争力。在来自6个模型家族、参数范围从2B到235B的11个开源VLM上，我们的发现表明，当前VLM作为符号推理器比作为直接视觉观察者更有效。通过将视频投影到语言领域，我们提供了一种可扩展、无需微调的替代端到端方法，更好地利用了这些模型的潜在推理优势。代码将公开。
英文摘要：
Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference alike. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
