arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.07448cs.CVcs.CL
收藏

文档图像分类器的分布外性能评估

Evaluating Out-of-Distribution Performance on Document Image Classifiers

DryvIQ(DryvIQ公司);University of Michigan(密歇根大学);Vanderbilt University(范德堡大学)

Stefan Larson, Gordon Lim, Yutong Ai, David Kuang, Kevin Leach

摘要:

文档分类器处理从不同于训练分布的分布中提取的输入的能力对于鲁棒部署和泛化性至关重要。RVL-CDIP语料库是文档分类的事实标准基准,但据我们所知,所有使用该语料库的研究都不包括对分发外文档的评估。在本文中,我们策划并发布了一个新的分发基准,用于评估文档分类器的分发性能。我们新的分发外基准由两种类型的文档组成:那些不属于16个域内RVL-CDIP类别(RVL-CDIP-O)中的任何一个的文档,以及那些属于16个域内类别之一但来自与原始RVL-CDIP数据集不同的分布的文档(RVL-CDIP-N)。虽然先前对域内RVL-CDIP文档进行文档分类的工作报告了高准确度分数,但我们发现这些模型在我们新的域外RVL-CDIP-N基准测试中表现出大约15-30%的准确度下降,并且进一步努力区分域内RVL-CDIP-N和域外RVL-CDIP-O输入。我们的新基准为研究人员提供了一个有价值的新资源,用于分析文档分类器的分布性能。我们新的分发外数据可以在https://github.com/gxlarson/rvl-cdip-ood上找到。

英文摘要:

The ability of a document classifier to handle inputs that are drawn from a distribution different from the training distribution is crucial for robust deployment and generalizability. The RVL-CDIP corpus is the de facto standard benchmark for document classification, yet to our knowledge all studies that use this corpus do not include evaluation on out-of-distribution documents. In this paper, we curate and release a new out-of-distribution benchmark for evaluating out-of-distribution performance for document classifiers. Our new out-of-distribution benchmark consists of two types of documents: those that are not part of any of the 16 in-domain RVL-CDIP categories (RVL-CDIP-O), and those that are one of the 16 in-domain categories yet are drawn from a distribution different from that of the original RVL-CDIP dataset (RVL-CDIP-N). While prior work on document classification for in-domain RVL-CDIP documents reports high accuracy scores, we find that these models exhibit accuracy drops of between roughly 15-30% on our new out-of-domain RVL-CDIP-N benchmark, and further struggle to distinguish between in-domain RVL-CDIP-N and out-of-domain RVL-CDIP-O inputs. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution performance on document classifiers. Our new out-of-distribution data can be found at https://github.com/gxlarson/rvl-cdip-ood.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑