arXiv 2605.147422026-05-14cs.CVcs.RO

EARL：一种统一的分析引导强化学习框架，用于第一人称交互推理与像素定位

EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR（香港理工大学电子与电气工程系）；Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR（香港理工大学新兴跨学科领域研究中心）

Yuejiao Su, Xinshen Zhang, Zhen Ye, Lei Yao, Lap-Pui Chau, Yi Wang

备注：: Accepted at ICML 2026. Project page: https://github.com/yuggiehk/EARL

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出EARL框架，通过分析引导特征合成器提升第一人称交互推理与像素定位的准确性，实验显示在Ego-IRGBench上像素定位达到65.48% cIoU，优于现有方法。

摘要：

理解人类与环境的交互从第一人称视觉是助行机器人和具身智能体的关键，但现有多模态大语言模型仍难以准确推理和细粒度像素定位。为此，本文引入EARL，一种第一人称分析引导强化学习框架，明确将粗粒度交互语义转移到查询导向的回答和定位。具体而言，EARL采用两阶段解析框架，包括粗粒度解释和细粒度响应。第一阶段整体解释第一人称交互并生成结构化文本描述。第二阶段根据用户查询生成文本答案和像素级掩码。为了连接两个阶段，我们提取全局交互描述作为语义先验，通过新颖的分析引导特征合成器（AFS）进行查询导向推理。为了优化异构输出，包括文本答案、边界框和定位掩码，我们设计多方面的奖励函数，并用GRPO训练响应阶段。实验表明，EARL在Ego-IRGBench上达到65.48% cIoU的像素定位，优于先前基于强化学习的方法，同时在EgoHOS上的OOD定位结果表明其在未见的第一人称定位场景中具有强泛化能力。

英文摘要：

Understanding human--environment interactions from egocentric vision is essential for assistive robotics and embodied intelligent agents, yet existing multimodal large language models (MLLMs) still struggle with accurate interaction reasoning and fine-grained pixel grounding. To this end, this paper introduces EARL, an Egocentric Analysis-guided Reinforcement Learning framework that explicitly transfers coarse interaction semantics to query-oriented answering and grounding. Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response. The first stage holistically interprets egocentric interactions and generates a structured textual description. The second stage produces the textual answer and pixel-level mask in response to the user query. To bridge the two stages, we extract a global interaction descriptor as a semantic prior, which is integrated via a novel Analysis-guided Feature Synthesizer (AFS) for query-oriented reasoning. To optimize heterogeneous outputs, including textual answers, bounding boxes, and grounding masks, we design a multi-faceted reward function and train the response stage with GRPO. Experiments on Ego-IRGBench show that EARL achieves 65.48% cIoU for pixel grounding, outperforming previous RL-based methods by 8.37%, while OOD grounding results on EgoHOS indicate strong transferability to unseen egocentric grounding scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。