OCRVerse: 向端到端视觉语言模型中的整体OCR迈进
OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
AI总结：
OCRVerse是一种端到端的全面OCR方法，通过多领域训练实现文本和视觉导向OCR的统一，提升跨领域数据处理能力。
摘要：
大型视觉语言模型的发展推动了对管理并应用大量多模态数据的需求，使OCR技术，即从视觉图像中提取信息，变得越来越受欢迎。然而，现有的OCR方法主要集中在从图像或扫描文档中识别文本元素（文本导向OCR），忽视了从视觉信息密集的图像源中识别视觉元素（视觉导向OCR），如图表、网页和科学图表。实际上，这些视觉信息密集的图像在互联网上广泛存在，具有显著的现实应用价值，如数据可视化和网页分析。在本技术报告中，我们提出了OCRVerse，这是首个以端到端方式实现的全面OCR方法，能够统一文本导向OCR和视觉导向OCR。为此，我们构建了全面的数据工程，涵盖了广泛文本导向文档，如报纸、杂志和书籍，以及视觉导向的渲染复合物，包括图表、网页和科学图表。此外，我们提出了一种两阶段SFT-RL多领域训练方法用于OCRVerse。SFT直接混合跨领域数据以训练并建立初始领域知识，而RL则专注于为每个领域的特征设计个性化奖励策略。具体而言，由于不同领域需要各种输出格式和预期输出，我们在RL阶段提供了足够的灵活性，以为每个领域定制灵活的奖励信号，从而提高跨领域融合并避免数据冲突。实验结果证明了OCRVerse的有效性，实现了在文本导向和视觉导向数据类型上的竞争性结果，甚至与大规模开源和闭源模型相当。
英文摘要：
The development of large vision language models drives the demand for managing, and applying massive amounts of multimodal data, making OCR technology, which extracts information from visual images, increasingly popular. However, existing OCR methods primarily focus on recognizing text elements from images or scanned documents (Text-centric OCR), neglecting the identification of visual elements from visually information-dense image sources (Vision-centric OCR), such as charts, web pages and science plots. In reality, these visually information-dense images are widespread on the internet and have significant real-world application value, such as data visualization and web page analysis. In this technical report, we propose OCRVerse, the first holistic OCR method in end-to-end manner that enables unified text-centric OCR and vision-centric OCR. To this end, we constructe comprehensive data engineering to cover a wide range of text-centric documents, such as newspapers, magazines and books, as well as vision-centric rendered composites, including charts, web pages and scientific plots. Moreover, we propose a two-stage SFT-RL multi-domain training method for OCRVerse. SFT directly mixes cross-domain data to train and establish initial domain knowledge, while RL focuses on designing personalized reward strategies for the characteristics of each domain. Specifically, since different domains require various output formats and expected outputs, we provide sufficient flexibility in the RL stage to customize flexible reward signals for each domain, thereby improving cross-domain fusion and avoiding data conflicts. Experimental results demonstrate the effectiveness of OCRVerse, achieving competitive results across text-centric and vision-centric data types, even comparable to large-scale open-source and closed-source models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
