arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.12358cs.CVcs.CL
收藏

ProcTag:用于评估文档指令数据效用的过程标记方法

ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data

College of Computer Science, Zhejiang University(浙江大学计算机科学学院);Alibaba Group(阿里巴巴集团)

Yufan Shen, Chuwei Luo, Zhaoqing Zhu, Yang Chen, Qi Zheng, Zhi Yu, Jiajun Bu, Cong Yao

AI总结:

本文提出ProcTag方法,通过对指令执行过程而非文本进行标记来评估文档指令数据效用,并结合DocLayPrompt提示策略,实现高效采样,仅需30.5%指令即可达到完整数据集100%的效用。

摘要:

近年来,大语言模型(LLMs)和多模态大语言模型(MLLMs)在文档视觉问答(VQA)任务上展现出有前景的结果,尤其是在文档指令数据集上训练之后。对于构建高效用的指令数据而言,一种有效的文档指令数据评估方法至关重要,这反过来又有助于训练用于文档VQA的LLMs和MLLMs。然而,现有的大多数指令数据评估方法仅限于指令本身的文本内容,从而阻碍了对文档指令数据集的有效评估,并限制了其构建。在本文中,我们提出ProcTag,一种面向数据的方法,用于评估文档指令数据的效用。ProcTag创新性地对指令的执行过程而非指令文本本身进行标记。通过利用这些标记的多样性和复杂性来评估给定数据集的效用,ProcTag能够对文档指令进行选择性采样或过滤。此外,我们提出DocLayPrompt,一种新颖的半结构化布局感知文档提示策略,用于有效表示文档。实验表明,使用ProcTag对现有的开源和生成的文档VQA/指令数据集进行采样,显著优于当前评估指令数据的方法。令人印象深刻的是,在生成的文档数据集中基于ProcTag进行采样时,仅需30.5%的文档指令即可达到与完整数据集相比100%的效用。代码已在https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag公开。

英文摘要:

Recently, large language models (LLMs) and multimodal large language models (MLLMs) have demonstrated promising results on document visual question answering (VQA) task, particularly after training on document instruction datasets. An effective evaluation method for document instruction data is crucial in constructing instruction data with high efficacy, which, in turn, facilitates the training of LLMs and MLLMs for document VQA. However, most existing evaluation methods for instruction data are limited to the textual content of the instructions themselves, thereby hindering the effective assessment of document instruction datasets and constraining their construction. In this paper, we propose ProcTag, a data-oriented method that assesses the efficacy of document instruction data. ProcTag innovatively performs tagging on the execution process of instructions rather than the instruction text itself. By leveraging the diversity and complexity of these tags to assess the efficacy of the given dataset, ProcTag enables selective sampling or filtering of document instructions. Furthermore, DocLayPrompt, a novel semi-structured layout-aware document prompting strategy, is proposed for effectively representing documents. Experiments demonstrate that sampling existing open-sourced and generated document VQA/instruction datasets with ProcTag significantly outperforms current methods for evaluating instruction data. Impressively, with ProcTag-based sampling in the generated document datasets, only 30.5\% of the document instructions are required to achieve 100\% efficacy compared to the complete dataset. The code is publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/ProcTag.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑