ANNEXE：用于自我中心交互的统一分析、回答与像素级定位
ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction
AI总结：
本文提出了自我中心交互推理与像素定位任务Ego-IRG及配套数据集Ego-IRGBench，并设计了统一的ANNEXE模型，利用多模态大语言模型同时生成文本与像素级输出，实现全面的自我中心交互理解。
摘要：
自我中心交互感知是研究人机交互的重要分支之一，为开发下一代智能系统奠定了基础。然而，现有的自我中心交互理解方法无法根据用户查询同时产生连贯的文本和像素级响应，这缺乏应对不同下游应用需求的灵活性。为了全面理解自我中心交互，本文提出了一项名为Egocentric Interaction Reasoning and pixel Grounding（Ego-IRG，自我中心交互推理与像素定位）的新任务。以带有查询的自我中心图像作为输入，Ego-IRG是首个旨在通过三个关键步骤解决交互的任务：分析、回答和像素定位，从而产生流畅的文本和细粒度的像素级响应。另一个挑战是，现有数据集无法满足Ego-IRG任务的条件。为了解决这一限制，本文在大量人工工作的基础上创建了Ego-IRGBench数据集，其中包含超过20k张自我中心图像、160万个查询以及关于交互的相应多模态响应。此外，我们设计了一个统一的ANNEXE模型，利用多模态大语言模型生成文本和像素级输出，从而实现对自我中心交互的全面解读。在Ego-IRGBench上的实验展示了我们的ANNEXE模型与其他工作相比的有效性。
英文摘要：
Egocentric interaction perception is one of the essential branches in investigating human-environment interaction, which lays the basis for developing next-generation intelligent systems. However, existing egocentric interaction understanding methods cannot yield coherent textual and pixel-level responses simultaneously according to user queries, which lacks flexibility for varying downstream application requirements. To comprehend egocentric interactions exhaustively, this paper presents a novel task named Egocentric Interaction Reasoning and pixel Grounding (Ego-IRG). Taking an egocentric image with the query as input, Ego-IRG is the first task that aims to resolve the interactions through three crucial steps: analyzing, answering, and pixel grounding, which results in fluent textual and fine-grained pixel-level responses. Another challenge is that existing datasets cannot meet the conditions for the Ego-IRG task. To address this limitation, this paper creates the Ego-IRGBench dataset based on extensive manual efforts, which includes over 20k egocentric images with 1.6 million queries and corresponding multimodal responses about interactions. Moreover, we design a unified ANNEXE model to generate text- and pixel-level outputs utilizing multimodal large language models, which enables a comprehensive interpretation of egocentric interactions. The experiments on the Ego-IRGBench exhibit the effectiveness of our ANNEXE model compared with other works.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
