在大规模视觉-语言模型中表示在线手写以进行识别
Representing Online Handwriting for Recognition in Large Vision-Language Models
AI总结：
本文提出一种同时包含时序笔画文本和图像的数字墨水标记化表示，使现成视觉-语言模型无需改架构即可在在线手写识别上达到或超越最先进水平。
摘要：
配备触摸屏和手写笔的平板电脑日益普及，其一项关键功能是将手写转换为文本，从而支持搜索、索引和人工智能辅助。与此同时，视觉-语言模型（VLMs）如今已成为图像理解的首选方案，这既得益于它们在各种任务上的最先进性能，也得益于统一的训练、微调和推理方法的简洁性。虽然VLMs在基于图像的任务上取得了高性能，但当以朴素方式应用时，即通过将手写渲染为图像并执行光学字符识别（OCR），它们在手写识别上表现不佳。在本文中，我们研究使用VLMs进行在线手写识别，超越朴素的OCR方法。我们提出了一种新颖的数字墨水（在线手写）标记化表示，该表示既包含按时间排序的笔画序列作为文本，也包含其作为图像的形式。我们表明，这种表示产生的结果可与最先进的在线手写识别器相媲美或更优。通过在多个公共数据集上使用两个不同的VLM系列所获得的结果，展示了其广泛的适用性。我们的方法可应用于现成的VLMs，不需要对其架构进行任何更改，并且既可用于微调，也可用于参数高效调优。我们进行了详细的消融研究，以确定所提出表示的关键要素。
英文摘要：
The adoption of tablets with touchscreens and styluses is increasing, and a key feature is converting handwriting to text, enabling search, indexing, and AI assistance. Meanwhile, vision-language models (VLMs) are now the go-to solution for image understanding, thanks to both their state-of-the-art performance across a variety of tasks and the simplicity of a unified approach to training, fine-tuning, and inference. While VLMs obtain high performance on image-based tasks, they perform poorly on handwriting recognition when applied naively, i.e., by rendering handwriting as an image and performing optical character recognition (OCR). In this paper, we study online handwriting recognition with VLMs, going beyond naive OCR. We propose a novel tokenized representation of digital ink (online handwriting) that includes both a time-ordered sequence of strokes as text, and as image. We show that this representation yields results comparable to or better than state-of-the-art online handwriting recognizers. Wide applicability is shown through results with two different VLM families, on multiple public datasets. Our approach can be applied to off-the-shelf VLMs, does not require any changes in their architecture, and can be used in both fine-tuning and parameter-efficient tuning. We perform a detailed ablation study to identify the key elements of the proposed representation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
