ECHO：面向仅腕部操作的事件增强上下文（事后与前瞻）

中文摘要 AI 辅助

基于学习且依赖RGB相机的操作策略在极端曝光条件下常因观测退化而性能受损。事件相机通过异步检测像素级强度变化来缓解这种退化，从而提供高动态范围。然而，其观测高度依赖于相机位置，因为固定相机会遗漏静态场景内容，而腕部安装的相机运动会导致先前访问过的区域离开视野。为解决这些时空局限性，我们提出了ECHO（事件增强上下文，含事后与前瞻），一种仅腕部的潜在世界动作模型，它将腕部事件编码为紧凑的运动表示，为策略推理提供时间和空间上下文。具体而言，ECHO利用预训练的事件编码器来解释帧间视觉特征的变化。其后事模块通过过去的事件流保留夹爪轨迹，作为可寻址的离相机上下文。同时，前瞻模块引入可学习的事件预视查询，并监督其预测未来动作的事件窗口，使策略能够预测即将发生的场景变化。在仅腕部的RLBench任务评估中，ECHO在正常光照下分别比RGB和RGB+事件基线高出20.6和12.0个百分点，在严重曝光下降情况下分别高出14.6和11.3个百分点，同时其性能也超越了使用第三视角相机的RGB参考方法。使用腕部安装事件相机的真实世界实验验证了ECHO在标称和严重黑暗光照下多个任务中均优于仅RGB和RGB+事件基线。项目页面位于此https URL。

英文摘要

Learning-based manipulation policies relying on RGB cameras often suffer from degraded observations under extreme exposure. Event cameras mitigate this degradation by asynchronously detecting pixel-level intensity changes to offer a high dynamic range. However, their observations heavily depend on camera placement, as fixed cameras miss static scene content while wrist-mounted camera motion causes previously visited regions to leave the field of view. To address these spatial-temporal limitations, we present ECHO (Event-augmented Context with Hindsight and Outlook), a wrist-only latent world action model that encodes wrist events into compact motion representations to provide temporal and spatial context for policy reasoning. Specifically, ECHO utilizes a pretrained event encoder to explain visual-feature changes between frames. Its hindsight module preserves the gripper trajectory with past event stream as addressable off-camera context. Concurrently, the outlook module introduces learnable event foresight queries supervised to anticipate the event window for future actions, enabling the policy to predict upcoming scene changes. Evaluated on wrist-only RLBench tasks, ECHO outperforms RGB and RGB+event baselines by 20.6 and 12.0 percentage points under normal lighting, and by 14.6 and 11.3 points under severe exposure drops, respectively, while also surpassing RGB references using a third-person camera. Real-world experiments with a wrist-mounted event camera validate that ECHO outperforms RGB-only and RGB+event baselines across multiple tasks under both nominal and severely dark lighting. Project page is at https://echo-wam.github.io/.

发表机构

• The Hong Kong University of Science and Technology（香港科技大学）
• The University of British Columbia（不列颠哥伦比亚大学）
• MMLab, The Chinese University of Hong Kong（香港中文大学多媒体实验室）
• Astribot
• ZENBOT

机构由 AI 辅助整理，请以论文原文为准。