OLIVE：对象级上下文视觉嵌入
OLIVE: Object Level In-Context Visual Embeddings
AI总结：
提出OLIVE方法，用上下文视觉对象向量提示大模型实现可控对象级推理，加速训练并支持区域级检索的零样本泛化。
摘要：
近期的通用视觉语言模型（VLMs）在多样的多模态任务中展现了令人印象深刻的推理能力。然而，这些模型在细粒度对象级理解和定位方面仍然存在困难。在建模方面，现有的VLMs隐式地将文本token与图像patch token对齐，这对于相同粒度的嵌入对齐是无效的，并且不可避免地引入了嘈杂的虚假背景特征。此外，这些模型在泛化到未见视觉概念时存在困难，若不进一步微调，可能无法可靠地用于特定领域任务。为了解决这些局限性，我们提出了一种新方法，用上下文视觉对象向量提示大型语言模型，从而实现可控的对象级推理。这消除了融合冗长图像patch特征序列的必要性，并显著加快了训练速度。此外，我们提出使用我们的对象表示进行区域级检索，促进无需额外训练即可快速适应新对象。我们的实验表明，我们的方法在指代对象分类和描述任务上取得了有竞争力的性能，同时提供了零样本泛化能力以及对视觉挑战性场景的鲁棒性。
英文摘要：
Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with fine-grained object-level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object-level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitating rapid adaptation to new objects without additional training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance, while also offering zero-shot generalization and robustness to visually challenging contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
