视觉语言模型能否在生产环境中取代基于OCR的视觉问答流水线？零售领域的案例研究
Can Visual Language Models Replace OCR-Based Visual Question Answering Pipelines in Production? A Case Study in Retail
AI总结：
本文在生产级零售场景下评估了多种开源与商业视觉语言模型在VQA和OCR任务中的表现，发现它们虽能准确识别品牌和价格，但在细粒度产品名称识别和折扣概念理解上存在明显局限。
摘要：
生产级视觉问答（VQA）任务的部署大多仍构建为包含图像预处理、目标与文本检测、光学字符识别（OCR）以及（主要为有监督的）目标分类等独立步骤的处理流水线。然而，视觉基础模型和视觉语言模型（VLMs）的最新进展引出了一个问题：这些定制训练的多步方法能否被预训练的单步VLMs所取代。本文在生产级场景下分析了多种VLMs在VQA和OCR任务中的性能与局限。使用Retail-786k数据集的数据，我们研究了预训练VLMs回答图像中广告产品详细问题的能力。研究包含两个商业模型GPT-4V和GPT-4o，以及四个开源模型：InternVL、LLaVA 1.5、LLaVA-NeXT和CogAgent。初步结果表明，开源与商业模型之间总体上没有巨大的性能差距。然而，我们观察到VLM性能存在强烈的任务依赖性差异：尽管大多数模型能高精度回答关于产品品牌和价格的问题，但它们同时完全无法正确识别具体的产品名称或折扣。这表明VLMs在解决细粒度分类任务以及对折扣这一更抽象概念进行建模方面存在问题。
英文摘要：
Most production-level deployments for Visual Question Answering (VQA) tasks are still build as processing pipelines of independent steps including image pre-processing, object- and text detection, Optical Character Recognition (OCR) and (mostly supervised) object classification. However, the recent advances in vision Foundation Models [25] and Vision Language Models (VLMs) [23] raise the question if these custom trained, multi-step approaches can be replaced with pre-trained, single-step VLMs. This paper analyzes the performance and limits of various VLMs in the context of VQA and OCR [5, 9, 12] tasks in a production-level scenario. Using data from the Retail-786k [10] dataset, we investigate the capabilities of pre-trained VLMs to answer detailed questions about advertised products in images. Our study includes two commercial models, GPT-4V [16] and GPT-4o [17], as well as four open-source models: InternVL [5], LLaVA 1.5 [12], LLaVA-NeXT [13], and CogAgent [9]. Our initial results show, that there is in general no big performance gap between open-source and commercial models. However, we observe a strong task dependent variance in VLM performance: while most models are able to answer questions regarding the product brand and price with high accuracy, they completely fail at the same time to correctly identity the specific product name or discount. This indicates the problem of VLMs to solve fine-grained classification tasks as well to model the more abstract concept of discounts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
