arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.02417cs.ROcs.CVcs.LG

LIME: 从自我中心视频学习意图感知的相机运动

LIME: Learning Intent-aware Camera Motion from Egocentric Video

ETH Zurich(苏黎世联邦理工学院);Microsoft(微软);University of Bonn(波恩大学)

Boyang Sun, Jiajie Li, Yung-Hsu Yang, Chenyangguang Zhang, Tim Engelbracht, Sunghwan Hong, Cesar Cadena, Marc Pollefeys, Hermann Blum

AI总结:

提出LIME模型,从自我中心视频中挖掘多意图相机运动监督,结合自回归观察增益输出与连续流匹配位姿头,实现语言条件相机运动生成。

摘要:

自主机器人通常需要在行动之前移动相机:检查物体、揭示被遮挡区域或获得符合用户意图的视图。虽然视觉-语言导航将指令转换为基础运动,视觉-语言-动作策略将指令映射到操作动作,但语言条件的相机运动作为一类动作仍相对未被充分探索。我们提出了语言条件相机运动生成:给定当前RGB观察和自由形式的自然语言意图,预测下一个观察的相对目标相机位姿。这项任务本质上具有挑战性:视角变化由潜在的感知意图驱动,有效的运动可能在不同语义粒度上操作,从进入房间到环顾角落、检查可见物体或揭示被遮挡的细节。为了建模这种结构,我们从自我中心视频中挖掘多意图相机运动监督,将合理的意图和观察增益描述与相对SE(3)目标位姿配对。我们提出了LIME,一种视觉-语言相机运动生成器,它结合了自回归观察增益输出和连续流匹配位姿头。这种设计使模型能够联合预测下一个视图应该揭示什么,同时表示多假设目标视图。在跨实验和下游机器人任务中,我们展示了LIME可以从被动人类视频中学习主动选择相机位姿,将普通的自我中心记录转化为意图感知主动感知的监督。

英文摘要:

Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, vision-language conditioned camera motion remains comparatively underexplored as a first-class action. We formulate vision-language conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. The challenge is that useful viewpoint changes depend on latent perceptual intent, ranging from coarse spatial moves to fine-grained inspection or occlusion revealing. To model this structure, we mine multi-intent camera-motion supervision from egocentric video via hindsight labeling, pairing each relative SE(3) target pose with plausible intents and observation-gain descriptions inferred from the reached view. We propose LIME, a vision-language camera-motion generator that combines an autoregressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across viewpoint-prediction experiments and downstream robotic tasks, we show that LIME learns intent-conditioned camera motion from passive egocentric video, turning ordinary human recordings into supervision for a reusable active-perception primitive that supports general-purpose viewpoint selection, manipulation, embodied question answering, and mid-distance navigation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑