面向视觉实体识别的语言模型基础对齐
Grounding Language Models for Visual Entity Recognition
AI总结：
提出 AutoVER 自回归模型，通过检索增强受限生成与困难负样本对比训练，在 Oven-Wiki 基准上显著提升了视觉实体识别的域内外及推理查询性能。
摘要：
我们提出了 AutoVER，一种用于视觉实体识别的自回归模型。该模型通过采用检索增强的受限生成，扩展了自回归多模态大语言模型。它缓解了在域外实体上的低性能问题，同时在需要视觉情境推理的查询中表现出色。该方法通过在序列到序列目标的同时，对困难负样本对进行对比训练，无需外部检索器即可学会在庞大标签空间中区分相似实体。在推理阶段，检索到的候选答案列表通过移除无效解码路径来显式引导语言生成。所提方法在最近提出的 Oven-Wiki 基准的不同数据集划分上取得了显著提升。Entity seen 划分上的准确率从 32.7% 提升至 61.5%。它还在 unseen 和 query 划分上以两位数的显著优势展现出卓越性能。
英文摘要：
We introduce AutoVER, an Autoregressive model for Visual Entity Recognition. Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation. It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning. Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever. During inference, a list of retrieved candidate answers explicitly guides language generation by removing invalid decoding paths. The proposed method achieves significant improvements across different dataset splits in the recently proposed Oven-Wiki benchmark. Accuracy on the Entity seen split rises from 32.7% to 61.5%. It also demonstrates superior performance on the unseen and query splits by a substantial double-digit margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
