从我的视角到你的视角：利用特权自我中心监督从外中心视频中学习自我中心线索
From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
- University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
- Elorian AI
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
研究针对视觉语言模型难以从外中心视频推断自我中心属性的问题，提出Ego2ExoVLM框架，利用时间同步视频对及特权监督，通过两个组件实现此能力，在多任务评估中取得领先性能。
AI中文摘要：
视觉语言模型（VLMs）在广泛的视频理解任务中取得了强大性能。然而，其视角不变的训练限制了从外中心视频观察中推断自我中心属性（如人与物体交互）的能力，这在日常生活活动（ADL）监测等应用中尤为关键。我们提出Ego2ExoVLM框架，在训练期间利用时间同步的自我-外中心视频对，使VLMs能直接从外中心视频推断自我中心属性。关键在于将自我中心视角视为特权监督。该框架由两个互补组件组成，并引入Ego-in-Exo Perception基准进行评估。在跨越该基准和现有ADL基准的10项任务上评估，Ego2ExoVLM在ADL-X基准套件上取得了领先性能。所有代码、模型和数据将在指定网址发布。
英文摘要：
Vision Language Models (VLMs) have achieved strong performance across a wide range of video understanding tasks. However, their viewpoint-invariant training limits their ability to infer egocentric properties, such as human-object interactions, from exocentric video observations. This limitation is particularly critical for applications such as Activities of Daily Living (ADL) monitoring, where understanding egocentric properties is essential but deploying wearable egocentric cameras is often impractical. We propose Ego2ExoVLM, a framework that enables VLMs to infer egocentric properties directly from exocentric videos by leveraging time-synchronized ego-exo video pairs during training. Our key insight is to treat the egocentric viewpoint as privileged supervision, providing rich interaction signals that are available only during training. Ego2ExoVLM consists of two complementary components: Ego2Exo Sequence Distillation, which transfers egocentric reasoning through a language-level sequence distillation objective, and Ego Adaptive Visual Tokens, which encourage the model to surface interaction-relevant cues within exocentric visual representations. To evaluate this capability, we introduce Ego-in-Exo Perception, a benchmark for assessing the understanding of egocentric properties from exocentric videos. We evaluate Ego2ExoVLM on 10 tasks spanning Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art performance on the ADL-X benchmark suite and consistently outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.
