SAKE: 为基于地面的多模态命名实体识别的自我意识知识利用-探索
SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition
AI总结:
针对开放世界社交媒体平台中长尾、快速演变和未见实体的挑战,SAKE提出一种端到端代理框架,通过自我意识推理和自适应搜索工具调用,结合内部知识利用和外部知识探索,提升多模态命名实体识别的精度与鲁棒性。
摘要:
基于地面的多模态命名实体识别(GMNER)旨在从图像-文本对中提取命名实体并定位其视觉区域,是各种下游应用的关键能力。在开放世界社交媒体平台上,GMNER仍然具有挑战性,因为存在长尾、快速演变和未见实体。为此,现有方法通常依赖于通过启发式检索进行外部知识探索或通过在多模态大语言模型(MLLMs)中进行迭代细化进行内部知识利用。然而,启发式检索往往会引入噪声或冲突的证据,从而降低对已知实体的精度,而仅靠内部利用则受限于MLLMs的知识边界并容易产生幻觉。为此,我们提出了SAKE,一种端到端的代理框架,通过自我意识推理和自适应搜索工具调用,和谐地结合内部知识利用和外部知识探索。我们通过一个两阶段训练范式实现这一目标。首先,我们提出了难度感知搜索标签生成,通过多次前向采样量化模型的实体层面不确定性,生成显式的知识缺口信号。基于这些信号,我们构建了SAKE-SeCoT,一个高质量的推理链数据集,通过监督微调使模型具备基本的自我意识和工具使用能力。其次,我们采用代理强化学习,使用混合奖励函数,惩罚不必要的检索,使模型从僵化的搜索模仿进化为真正的自我意识决策,即何时真正需要检索。在两个广泛使用的社交媒体基准测试中进行了大量实验,证明了SAKE的有效性。
英文摘要:
Grounded Multimodal Named Entity Recognition (GMNER) aims to extract named entities and localize their visual regions within image-text pairs, serving as a pivotal capability for various downstream applications. In open-world social media platforms, GMNER remains challenging due to the prevalence of long-tailed, rapidly evolving, and unseen entities. To tackle this, existing approaches typically rely on either external knowledge exploration through heuristic retrieval or internal knowledge exploitation via iterative refinement in Multimodal Large Language Models (MLLMs). However, heuristic retrieval often introduces noisy or conflicting evidence that degrades precision on known entities, while solely internal exploitation is constrained by the knowledge boundaries of MLLMs and prone to hallucinations. To address this, we propose SAKE, an end-to-end agentic framework that harmonizes internal knowledge exploitation and external knowledge exploration via self-aware reasoning and adaptive search tool invocation. We implement this via a two-stage training paradigm. First, we propose Difficulty-aware Search Tag Generation, which quantifies the model's entity-level uncertainty through multiple forward samplings to produce explicit knowledge-gap signals. Based on these signals, we construct SAKE-SeCoT, a high-quality Chain-of-Thought dataset that equips the model with basic self-awareness and tool-use capabilities through supervised fine-tuning. Second, we employ agentic reinforcement learning with a hybrid reward function that penalizes unnecessary retrieval, enabling the model to evolve from rigid search imitation to genuine self-aware decision-making about when retrieval is truly necessary. Extensive experiments on two widely used social media benchmarks demonstrate SAKE's effectiveness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。