arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.18706cs.CV
收藏

The Socface 项目:一个世纪法国人口普查的大规模收集、处理与分析

The Socface Project: Large-Scale Collection, Processing, and Analysis of a Century of French Censuses

TEKLIA(泰克利娅公司);Institut National d’Etudes Démographiques (INED)(法国国家人口研究院);Paris School of Economics (PSE)(巴黎经济学院)

Mélodie Boillet, Solène Tarride, Manon Blanco, Valentin Rigal, Yoann Schneider, Bastien Abadie, Lionel Kesztenbaum, Christopher Kermorvant

AI总结:

本文提出针对1836-1936年法国人口普查名单的完整处理工作流,涵盖大规模数据收集、协作标注、模型训练与批量处理,并证明单一表格识别模型可处理超45万张图像。

摘要:

本文提出了一个完整的处理工作流,用于从1836年至1936年的法国人口普查名单中提取信息。这些名单包含居住在法国的个人及其家庭的信息。我们旨在利用自动手写表格识别技术提取这些表格中包含的所有信息。在 The Socface 项目结束时(我们的工作正是在该项目中进行的),提取的信息将被重新分发给各省级档案馆,并且具名名单将向公众免费开放,使任何人都能浏览数亿条记录。提取的数据将被人口学家用于分析社会随时间的变迁,从而显著提升我们对法国经济和社会结构的理解。针对此项目,我们开发了一套完整的处理工作流:从法国省级档案馆进行大规模数据收集、文档的协作标注、手写表格文本与结构识别模型的训练,以及数百万图像的批量处理。我们展示了为轻松收集和处理数百万页而开发的工具。我们还证明,使用单一的表格识别模型处理如此种类繁多的表格是可行的,该模型利用整页图像来识别个人信息、对其进行分类,并自动将他们分组为家庭。整个流程已成功用于处理一个省级档案馆的文档,涵盖超过450,000张图像。

英文摘要:

This paper presents a complete processing workflow for extracting information from French census lists from 1836 to 1936. These lists contain information about individuals living in France and their households. We aim at extracting all the information contained in these tables using automatic handwritten table recognition. At the end of the Socface project, in which our work is taking place, the extracted information will be redistributed to the departmental archives, and the nominative lists will be freely available to the public, allowing anyone to browse hundreds of millions of records. The extracted data will be used by demographers to analyze social change over time, significantly improving our understanding of French economic and social structures. For this project, we developed a complete processing workflow: large-scale data collection from French departmental archives, collaborative annotation of documents, training of handwritten table text and structure recognition models, and mass processing of millions of images. We present the tools we have developed to easily collect and process millions of pages. We also show that it is possible to process such a wide variety of tables with a single table recognition model that uses the image of the entire page to recognize information about individuals, categorize them and automatically group them into households. The entire process has been successfully used to process the documents of a departmental archive, representing more than 450,000 images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑